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Achieve is an independent, bipartisan, nonprofit organization created by governors and 
corporate leaders to help states and the private sector raise standards and performance in 
Americans schools. Founded at the 1996 National Education Summit, Achieve has spon- 
sored two additional Summits in 1999 and 2001. 



Achieve helps states raise academic standards, measure performance against those standards, 
establish clear accountability for results and strengthen public confidence in our education 
system. To do this, we: 



■ help states benchmark their standards, assessments and accountability systems against 
the best in the country and the world; 



I provide sustained public leadership and advocacy for the movement to raise stan- 
dards and improve student performance; 



I build partnerships that allow states to work together to improve teaching and learning 
and raise student achievement; and 



■ serve as a national clearinghouse on education standards and school reform. 
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Executive Summary, Part 11 



Achieve, Inc., was created after the 1996 National Education Summit by governors and business 
leaders to help states raise academic standards, improve student achievement and build support 
for-improving public education. As part of its mission. Achieve provides states with candid 
feedback on the quality of their academic standards, assessments, accountability systems and 
other policies to promote high academic achievement. To date. Achieve has worked with 16 
states in support of standards-based education reform. 

At the request of the Oklahoma Business and Education Coalition (OBEC), the Oklahoma State 
Regents for Higher Education, the Oklahoma State Department of Education and the Governor’s 
Office, Achieve examined Oklahoma’s system of academic standards and assessments as part of 
a broader review of the state’s education reform. In particular. Achieve examined Oklahoma’s 
Priority Academic Student Skills (PASS), comparing them to exemplary standards from other 
states and nations. Because ACT’s Educational Planning and Assessment System has played a 
pivotal role in helping prepare Oklahoma students for success in continuing education and 
careers in the New Economy, Achieve also reviewed ACT’s Standards for Transition and their 
alignment to Oklahoma standards. 

Having high-quality standards in place is critical to education reform; equally important is 
having challenging tests that measure whether students meet the standards. Achieve also 
conducted an alignment review of Oklahoma’s tests in English language arts and mathematics 
for grades 5 and 8; its high school English test; its field-test version of its Algebra I test; and 
ACT grade 11/12 assessments in English, reading and mathematics, comparing the content of 
each test, in turn, to the expectations described in the state standards. 

This report summarizes Achieve ’s findings and provides policymakers with answers to the 
following questions: 

• How do Oklahoma’s standards in English language arts and math compare with those of 
exemplary states and nations, and how well do ACT’s Standards for Transition align with 
Oklahoma’s standards? Are the expectations for students rigorous yet reasonable? 

• How well do Oklahoma’s tests in English language arts and math assess the knowledge 
and skills described in the standards? How challenging are the tests? 

• How well do ACT’s Standards for Transition and tests in English, reading and math 
match Oklahoma’s standards and tests? 
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Results for Oklahoma 



With the K-12 and higher education systems working in partnership, Oklahoma has made 
substantive progress in implementing a system of standards and tests. Achieve’s major findings 
are: 

Oklahoma’s standards in both English language arts and mathematics are 
comprehensive, written in clear prose and typically expressed in measurable terms. 

Oklahoma’s PASS contains the essential knowledge and skills necessary for students to become 
proficient in reading, writing and math. The grade-level placement of content is, in general, quite 
defensible. Reviewers commented that certain aspects of Oklahoma’s standards compare 
favorably to Achieve’s benchmark standards. For example, K-3 standards in English language 
arts provide comprehensive coverage of essential areas, while K-5 expectations in math la> a 
strong conceptual foundation for further study, balancing the attention given to procedures, 
concepts and application. 

There are other praiseworthy features of Oklahoma’s standards. They are written in jargon-free 
language that is accessible to the public. The standards — with a few notable exceptions — 
focus on the results of student learning that can be readily observed or tested. In practice, this 
means the standards emphasize concrete performances, such as “explain” or “analyze,” while 
avoiding those that focus oh the process of learning, such as “explore” or “investigate.” 

Oklahoma’s Core Curriculum Tests in English language arts and mathematics 
generally align well with those standards identified as appropriate for testing at the 
state level. 

In English language arts, reviewers reported that most test items are strongly or partially aligned 
to Oklahoma’s standards. They also found that the reading passages in the state’s tests meet the 
expectations of the standards in that they are keyed to the respective grade levels, hold the 
reader’s interest, and represent an excellent balance of informational text and fiction (this stands 
in sharp contrast to a number of the state tests reviewed by Achieve in which the choice of 
passages was disappointing). In addition, Oklahoma’s decision to administer tests that directly 
measure students’ ability to respond to the totality of a writing task — identifying audience and 
purpose; organizing the piece; and attending to grammar, syntax and punctuation — is sound, 
providing a more complete picture of a student’s writing proficiency than if the student merely 
responded to a series of multiple-choice questions about writing or editing. 

In math, reviewers similarly found that all of the items on Oklahoma’s tests are either strongly 
aligned or partially aligned to the state’s standards. 
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Oklahoma’s standards and tests lack a consistent level of rigor. 

Despite the strengths noted above, Oklahoma’s standards do not appear to be as rigorous as those 
of the benchmark standards. Oklahoma’s standards sometimes suffer from a lack of specificity, 
and this lack of precision makes it difficult to be sure of the depth of understanding required. 
Oklahoma tends to underspecify writing skills and overspecify reading skills, but without 
providing the clarity that comes from the use of suggested reading lists or sample text to 
illustrate the expected level of rigor. In addition, the English language arts standards do not 
provide the kind of clear and consistent progression of knowledge and skills characteristic of 
rigorous standards. In math, the lack of sample tasks makes it difficult to be sure of the rigor the 
state expects, especially in the area of mathematical process skills. 

Achieve also found that the level of rigor on the Oklahoma Core Curriculum Tests fluctuates. In 
English language arts, it is appropriate for grade 5, too low for grade 8 and nearly at the 
appropriate level on the English II test. In math, the level of rigor of the Algebra I field test is on 
target, but the tests in grades 5 and 8 sometimes measure standards from earlier grades and 
emphasize procedural knowledge at the expense of more intellectually demanding skills. 

Although act’s standards and Oklahoma’s standards are fundamentally compatible, 

ACT tests in English, reading and mathematics are not strongly aligned to Oklahoma’s 

standards. 

Oklahoma’s PASS is much broader than ACT’s Standards for Transition and is meant to 
describe a comprehensive set of student outcomes that reflect a whole K-12 curriculum. In 
contrast, ACT’s Standards for Transition are derived from ACT tests, focus on middle and high 
school knowledge and skills essential for college readiness, and are not designed to match a 
particular set of state standards. A comparison of the two standards documents reveals a few key 
differences: In English language arts, ACT’s standards are stated much more precisely and with 
greater rigor than are Oklahoma’s standards. In math, ACT’s standards tend to organize and 
emphasize content differently than do Oklahoma’s standards. 

Given the differences in standards, it is not surprising that ACT tests are not completely 
consistent with Oklahoma’s standards. After correcting for the fact that the ACT is aimed at late 
high school while Oklahoma’s tests are geared to early high school, some contrasts are evident. 
The reading section of the ACT contains passages that are more cognitively demanding than 
those contained in Oklahoma’s English II test, and ACT test questions also tend to demand more 
complex thinking skills than Oklahoma’s English II test. The English section of the ACT 
measures advanced knowledge of conventions and aspects of writing not addressed explicitly in 
Oklahoma’s standards. ACT math tests assess knowledge and skills found in Oklahoma’s 
standards but measure only a subset of the total content knowledge described in Oklahoma’s 
standards. To be specific, less than 40 percent of the content knowledge in Oklahoma’s courses 
in Algebra I, Geometry and Algebra II is assessed by the ACT. 
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Recommendations for Moving Forward 



Oklahoma is in the process of revising its standards, field testing its high school Algebra I test 
and extending its direct writing assessment to the 10^'’ grade. The timing is such that the state has 
an opportunity to build on the work it already has accomplished and go much further, in part by 
implementing the following recommendations: 

Make substantive revisions to PASS in English language arts and fine-tune PASS in 
mathematics. 

Oklahoma’s English language arts standards would benefit from reconstruction with an eye 
toward accomplishing these four essential improvements: 

1. Establish a clear organizational structure and focus to eliminate redundancies. 

2. Signal the most important content at each grade level. 

3. Clarify the level of rigor expected by including sample text or suggested reading lists. 

4. Describe a clear progression of knowledge and skills from grade to grade, particularly in 
writing, where the present standard does not provide an adequate scaffold for developing 
student proficiency over time. 

Oklahoma’s standards in math are stronger than those in English language arts but could be 
strengthened by compacting the treatment of arithmetic, which now spills from grades 1 through 
5 into grades 6 through 8, and the treatment of linear equations, which also spreads across too 
many grade levels. Consideration also should be given to expanding the treatment of specific 
topics in algebra and in probability, statistics and data analysis. In addition, the inclusion of 
sample problems would go a long way toward helping teachers and parents understand the level 
of proficiency the state expects. 

Oklahoma should stay the course with its plan to develop curriculum materials aligned to PASS 
and make them available widely using technology. 

Bring Oklahoma’s system of standards and tests into closer alignment with ACT’s 
standards and tests by judiciously augmenting the state system with key elements from 
the ACT. 

Oklahoma has used ACT’s standards and tests to good advantage, making them an integral part 
of state efforts to improve the academic preparation of middle and high school seniors. However, 
as it now stands, students who score well on Oklahoma’s English II test might find themselves 
struggling to achieve a comparable score on the ACT. They may not be sufficiently prepared for 
the complexity of text the ACT presents or the subtlety of the accompanying questions. To help 
close the gap, Oklahoma should consider including in its English II test at least one “stretch” 
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passage — aimed at higher performing 10* graders — and accompanied by questions that assess 
the more intellectually demanding parts of Oklahoma’s standards. In the area of writing, 
Oklahoma should consider incorporating the more demanding and varied skills measured by the 
ACT into PASS. 

Discrepancies between Oklahoma’s and ACT’s systems in math can be alleviated by infusing 
and assessing key topics included on the ACT but not contained in Oklahoma’s standards for 
Algebra I, Geometry and Algebra II. At the very least, it is important to highlight differences 
between the systems and communicate them to teachers, students and parents. 
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Introduction: Raising Standards in America’s Schools 



Since the release of A Nation at Risk in 1983, schools, states and national policymakers have 
been concerned with improving the level of academic achievement of all students. At that time, 
most school systems awarded diplomas based on Carnegie units, which generally represented 
“seat time,” as opposed to a demonstration of knowledge and skills. Under the Carnegie or 
credit-based system, postsecondary institutions and employers had difficulty determining what 
students had learned, since course content varied from school to school. Additionally, it was 
increasingly clear that a large percentage of our nation’s students, particularly minorities and the 
poor, were being underserved dramatically by their education system because they were not 
provided with the same rich curriculum and learning opportunities as their wealthier counterparts 
in the suburbs. 

In an attempt to raise the level of student learning across the board and create a more publicly 
accountable education system, states, districts and national organizations began the process of 
drafting content standards to define what students should know and be able to do. In Oklahoma 
and across the country, standards now are the driving force in efforts to improve equity and 
excellence in education by holding all students to common, high expectations. 

Since the early 1990s, 49 states have developed academic standards for their students, 48 states 
are putting in place assessments to measure those standards, and most are holding schools and 
students accountable for their performance on the assessments. By stating clearly the knowledge 
and skills students are expected to gain as a result of their schooling, reformers hope that 
students will better understand what is expected of them, schools will improve their programs to 
help students achieve those expectations, and low-performing districts and schools in particular 
will be challenged to raise the level of teaching and learning. Because states have made 
substantial investments in the new standards and tests — and because the accountability systems 
make these standards and assessments matter — policymakers and the public want to know how 
their standards compare to what other states and countries expect. They also want to know if the 
testing and accountability systems are as strong as those of successful states and whether they 
will accomplish their goals of helping all schools and students reach high standards. Achieve was 
created precisely to address these issues. 

Bom out of the 1996 National Education Summit, Achieve helps states ensure that they have in 
place standards that compare favorably with the academic expectations of other states and high- 
performing nations; assessments that accurately measure student achievement against those 
standards; and policies that provide appropriate information, incentives and support for schools 
and the public. An independent, bipartisan, nonprofit organization overseen by a board of 
directors composed of governors and corporate CEOs, Achieve serves as a clearinghouse and 
resource center on education standards, testing and accountability, working primarily with states 
to support their work in these areas. 
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The United States has come a long way in the effort to improve schools for all students through 
the development and implementation of standards-based reform. In the initial stage of the 
standards movement, states found it challenging to develop high-quality standards. The nation 
now has a better picture of what strong academic standards look like. They are clear and specific 
enough to guide curriculum planning and test development without infringing on local control. 
They set rigorous, yet reasonable, expectations for all students and raise the bar higher than it is 
currently set for many students. They integrate content knowledge with important thinking skills 
and learning processes. And they are widely read and understood by parents, educators, business 
people and policymakers. State assessments are evolving as well. More attention is being given 
to alignment; to incorporating a mix of formats (multiple-choice, short-answer and open-ended 
items), and to sharing concrete information with district educators and the general public. 

States also are revisiting the issue of local control, with each state striking a different balance. 
States are realizing that respecting local control, while ensuring high standards for all students, 
means emphasizing the knowledge and skills students are expected to learn and assisting 
educators by providing tools and training. This new understanding is at the heart of standards- 
based reform. The previous state role of monitoring compliance to rules and regulations has 
shifted to one of setting expectations for results, placing consequences on those results, and 
supporting schools and districts in meeting those expectations. 

The 2001 reauthorization of the Elementary and Secondary Education Act — the No Child Left 
Behind Act requires all states to examine their education systems to see if they can achieve 
the purpose stated in the law: helping all students to reach proficiency in reading and math within 
12 years. As states and school districts redesign their testing and accountability systems to 
comply with the new requirements, they will grapple with a host of thorny issues, including: 

• strengthening teacher preparation, certification and professional development; 

• developing or identifying materials to support state standards and assessments; 

• setting fair and defensible promotion and graduation requirements; 

• providing additional funding and programs for struggling students; 
e ensuring equitable treatment of special populations; and 

• establishing genuine accountability for all education stakeholders and policymakers. 

Benchmarking to the Best 

To help states such as Oklahoma in their efforts to continuously improve schooling for all 
students. Achieve provides a benchmarking service. Through benchmarking. Achieve compares 
a state’s practices to the best available models from the United States and the world. States such 
as Oklahoma that have sought benchmarking services from Achieve are committed to raising 
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standards for student performance and to holding schools accountable. These states want their 
citizens to know that the standards they have set compare favorably to the expectations other 
states and nations have for their students. They also want to understand whether the tests states 
use to assess student progress against the standards truly measure what they expect all students to 
know and be able to do. They want to know whether their accountability policies provide the 
right balance of pressure and support. Lastly, they want objective, credible, concrete 
recommendations for ways to improve their standards, assessments and accountability systems. 

Benchmarking is a highly respected practice in the business world. It is an activity that looks 
outward to find best practices and high performance and then measures actual business 
operations against those goals. Benchmarking in education follows the same principle. It is 
appropriate at a time when state education reforms are focused on raising student and school 
performance, as states want and need an external yardstick to gauge their efforts. 

By benchmarking academic standards, assessments and accountability policies. Achieve hopes to 
help states answer the following questions: 

• How do the state’s education standards compare with those of other high-performing 
states and nations? Are the expectations for students and schools high enough? 

• How well do the state’s assessments measure the knowledge and skills laid out in the 
standards? 

• Do the state’s accountability policies provide appropriate incentives for students and 
educators? 

Achieve is involved in benchmarking for another important reason: States traditionally have had 
limited access to high-quality, trustworthy information about education policies and practices. 
This is due partly to the fact that the standards movement in education is relatively young. But it 
also is a result of the disparate nature of much of the work that has been done to date. While the 
standards reviews and “report cards” issued by other organizations have helped to focus national 
attention on the quality of standards, their judgments often have been in conflict, and their tone 
has not always been constructive. States increasingly are looking for independent, credible 
advice on these issues. 

Achieve’ s benchmarking efforts are not designed to grade or rank states. Instead, we have 
created a service that is diagnostic in nature — one that yields detailed, reliable information that 
Achieve hopes states will find useful. In addition, Achieve’s focus on the policy enviromnent in 
states allows us to determine whether the standards are used as effectively as they could be and 
to suggest ways that state leaders could improve their practices so that they accomplish what 
they set out to do: to improve education performance for all young people. 
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The Achieve Benchmarking Methodology 



Achieve’s benchmarking and alignment methodology has been developed and tested over four 
years. Achieve staff and consultants have reviewed dozens of standards and tests and bring that 
experience to bear in applying these tools to Oklahoma’s standards and tests. 

Standards Benchmarking 

Achieve compares a state’s standards to state, national and international benchmark standards 
recognized for their quality and/or for producing high student achievement. This comparison of 
state standards to benchmark standards is designed to answer the following questions: 

• Do the standards define a comprehensive, yet manageable, academic core for all 
students? Are there key concepts or skills missing in the standards? 

• Do the standards convey both the breadth and depth of knowledge and skills expected of 
students? Have choices been made about what is most important for students to learn and 
when? 

• Are the standards as rigorous as they should be? Do the expectations described by the 
state standards compare favorably to those of the benchmark standards? 

• Do the standards define both what students should know and what they should be able to 
do with that knowledge? For example, are geometric concepts delineated while reasoning 
and problem-solving skills are developed as well? Or is one overemphasized at the 
expense of the other? 

• Are the standards clear and explicit, conveyed in a way that educators and parents can 
understand and use to improve student achievement? 

To ensure that the benchmark standards documents used as exemplars are indeed the best for this 
purpose. Achieve commissioned in 1999 expert reviews of a variety of sets of standards. 
Reviewers concluded that California’s and Massachusetts’ standards in English; those of North 
Carolina, Texas and New Standards in early literacy; and those of Arizona and Japan in math had 
the most value for benchmarking.’ 

Selecting these benchmarks proved to be a difficult task because no one set of standards is 
perfect, and judgments about the quality of standards are in some ways subjective. Still, Achieve 
is confident that the choices used in its current work reflect some of the best thinking from 



1 

Achieve created “benchmark profiles” for each of these documents that provide contextual information about the 
standards and summarize their strengths and weaknesses. The profiles are available upon request. 
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around the country, and that a careful comparison of a state’s standards to these benchmarks will 
yield helpful diagnostic information and policy suggestions for states to consider. 

For example, in the area of language arts, California distinguishes those standards concerned 
with informational text from those concerned with literature. This makes it easy for teachers to 
help students grasp the different strategies used in reading, understanding and writing the two 
types of text. Massachusetts organizes its standards in grade spans of two years, as do a number 
of states, but also articulates its expectations for student learning more precisely than do most 
states. In math, Arizona’s standards provide a level of specificity and detail that is helpful for 
teachers — especially for elementary generalists — while Japan’s standards provide an 
economical, yet focused and rigorous treatment of the discipline. 

Assessment-to-Standards Alignment Analysis 

Achieve’s assessment-to-standards analysis is designed to address the alignment of tests to 
standards. It helps uncover answers to the following issues: 

• Fairness. Does each assessment measure only content and skills reflected in the 
standards? Or, put differently, can everything on the test be found in the state standards? 

• Balance. Does each assessment measure the breadth and depth of content and skills in 
the standards? In other words, to what extent does each assessment measure the key 
content and skills for a grade level? 

• Rigor. Overall, is each assessment sufficiently challenging for students? Do the 
assessments grow more sophisticated from grade to grade? 

Alignment is not a “yes or no” question — nor is it a mathematical calculation. It is the extent to 
which standards and assessments are in agreement and serve in conjunction with one another to 
guide and support student learning. Consequently, responding to the above questions requires a 
systematic procedure to probe the different factors that contribute to alignment. Achieve’s 
process, or “protocol,” for determining the alignment of assessments to standards is based upon 
four criteria. Application of the protocol provides rich information about alignment of tests and 
standards, which typically is unavailable to states. The fundamental criteria for Achieve’s 
alignment process are: 

® Confirmation or construction of test blueprint. Reviewers check to see that each item 
corresponds to at least one standard or objective. If no test blueprint is provided by the 
state or testing company, then Achieve’s reviewers construct one. If the blueprint 
supplied by the test developer does not stand up to scrutiny (i.e., a significant number of 
items mapped to one standard or objective are found to be more closely related to a 
different one), reviewers may decide to revise the blueprint. 
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• Content centrality. This criterion examines the quality of the match between the content 
of each test item and the content of the related standard. Reviewers determine how 
closely the content of the item matches that of the related standard and then assign the 
item to one of four categories based on degree of alignment, from “not aligned” to 
“clearly aligned.” 

• Performance centrality. Each item places a certain type of cognitive demand on a 
student (e.g., the student is asked to “identify” or “analyze”). If an item simply requires a 
student to “identify” and the corresponding standard requires a student to “analyze,” then 
there is a mismatch between the two performances. Reviewers assign each item to one of 
four categories based on the degree of alignment, from “not aligned” to “clearly aligned.” 

• Challenge. This criterion is applied to both individual items and to the set of items that 
measure an entire strand, such as Measurement. Its purpose is to determine whether doing 
well on these items requires students to master challenging subject matter. At the item 
level, reviewers consider two factors related to challenge: source of challenge and level of 
cognitive demand. At the item set level, reviewers consider the overall level of challenge 
of the items mapped to a strand. 

o Source of challenge. This criterion attempts to uncover whether an individual test 
item is “fair.” Reviewers analyze whether an item is difficult because of the 
knowledge and skills it targets or for other reasons not related to the subject matter, 
such as relying unfairly on students’ background knowledge. They rate each item as 
having an appropriate or inappropriate source of challenge. Any item judged to have 
an inappropriate source of challenge is not examined when item sets are evaluated 
for level of challenge. 

o Level of cognitive demand. This criterion focuses on the type and level of thinking 
and reasoning required by the student for a particular item.^ A Level 1 (recall) item 
requires recall of information such as fact, definition, term or simple procedure. A 
Level 2 (skill/concept) item calls for engagement of some mental processing beyond 
a habitual response, with students required to make some decisions as to how to 
approach a problem or activity. Level 3 (strategic thinking) items require students to 
reason, plan or use evidence. And Level 4 (extended thinking) items require complex 
reasoning, planning, developing and thinking, typically over an extended period of 
time. 

o Level of challenge. This criterion applies to the set of items that maps to a standard. 
Reviewers compare the overall demand encompassed by a set of items to the level of 
demand expressed in the standard itself In addition to evaluating alignment, 
reviewers also judge whether the set of test items has a span of difficulty appropriate 



^ Norman L. Webb. (2001) Levels for Determining Depth of Knowledge. CCSSO TILSA Alignment Study, Version 
2.0, May 21-24, 2001. 
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for students at a given grade level. This judgment is based on the standards, the 
assessment and supporting materials such as student responses. Reviewers write a 
succinct summary of the level of challenge of each item set. 

o Balance and range. Balance compares the extent to which the knowledge and skills 
delineated in the standards receive the same emphasis on the assessment and 
determines if that emphasis is appropriate. Range is a quantitative check on the 
degree to which the items sample the knowledge and skills described in the standards 
since it is very difficult for one assessment to measure the full complement of 
knowledge and skills required by state standards. Evaluating balance and range 
provides both qualitative and quantitative information about the choices states or test 
developers have made. 

Working with Oklahoma 

Every three years, the State Board of Education is required by law to review and revise 
Oklahoma’s core curriculum. The documents analyzed by Achieve represent the 1999-2000 
review and revision of Oklahoma’s core curriculum, PASS, as coordinated and compiled by the 
State Department of Education. Due to legislative changes (House Bill 1759) in the graduation 
requirements and curriculum reviews by the American Federation of Teachers, Oklahoma 
Education Association and state content associations, major revisions were made to most 
curriculum areas. Kindergarten, Language Arts, and Reading standards were completely revised. 

The Oklahoma State Regents for Higher Education has supported the use of ACT’s Educational 
Planning and Assessment System (EPAS) since 1993, and ACT assessments for grades 1 1 and 12 
is a key component of this system. Accompanying EPAS is a set of statements that describe what 
students who score in various score ranges are likely to know and be able to do. These Standards 
for Transition reflect a progression of skills across score ranges and are organized by curricular 
strands. Since these Standards for Transition are important tools being used by Oklahoma 
educators to prepare high school students for postgraduate work or school. Achieve ’s reviewers 
were asked to review them, in addition to Oklahoma’s PASS for English language arts and math. 
The focus of the review is to look not only at the substance and quality of the Standards for 
Transition, but also at whether and how well they align with Oklahoma’s state standards. 

Achieve’s benchmarking analysis of PASS, comparison of ACT’s Standards for Transition with 
PASS, and alignment study of Oklahoma tests and ACT assessments to PASS occurred in 
multiple phases. In the first phase. Achieve senior staff examined Oklahoma’s PASS in English 
language arts and math standards against our benchmark standards, systematically comparing the 
content and skills found in PASS to those of Massachusetts and California in English language 
arts and to those of North Carolina, Texas and New Standards in early literacy. In math the 
comparison was made to Arizona, Japan and Achieve’s MAP expectations for the end of grade 8. 
In the second phase, national experts in English language arts and math standards reviewed 
PASS and matched ACT’s Standards for Transition to PASS, using comparisons to the 
benchmark standards and responding to a set of guiding questions, organized according to 
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criteria that distinguish exemplary state standards from marginal ones. The national experts — 
who hold diverse opinions about content, curriculum and assessment issues and have 
considerable experience in writing, researching and analyzing standards — submitted 
independent reviews of the standards to Achieve. 

In a third and separate process. Achieve compared Oklahoma’s state assessments in grades 5 and 
8 in English language arts and math, its high school English test, and its field-test version of the 
Algebra I test, as well as Form 57B of the ACT 1997-98 grade 1 1/12 assessments in English, 
reading and math to Oklahoma’s standards. 

To gauge the alignment of Oklahoma’s tests and ACT assessments to PASS, Achieve convened 
English language arts and math teams to examine the tests carefully and apply the Achieve 
alignment protocol. Achieve’ s senior consultants led their teams through a stepwise application 
of the protocol, with the goal of arriving at consensus judgments for each of the alignment 
criteria. 

Achieve’s alignment analysis is a process of managing expert judgment. There is no 
mathematical formula for matching a test to standards. Rather, the process relies on experienced, 
knowledgeable educators who bring that experience and knowledge to bear in applying the 
criteria for judging alignment. 

The reviewers are a deliberate mix of classroom teachers, curriculum specialists and subject- 
matter experts, each with extensive expertise in content and assessment design. They often have 
experience in large-scale assessments and/or standards development; represent a diversity of 
viewpoints on curriculum matters; and have worked in a variety of challenging school 
environments in rural, suburban and urban settings. This diversity of backgrounds has proven 
invaluable in arriving at considered judgments (for example, determining the appropriateness of 
a test item for a particular grade level). 

In writing this report. Achieve synthesized three reviews of Oklahoma’s English language arts 
and math standards and ACT’s Standards for Transition, highlighting the strengths and 
weaknesses of the documents as identified by the subject-area experts. Achieve also summarized 
the results of the alignment studies as reported by the review teams in English language arts and 
math. The findings described in this report represent consensus opinions of Achieve’s 
consultants and experts, but final judgments and conclusions rest with Achieve. 

It is important to keep in mind that Achieve’s “standards for standards” are very high; its 
recommendations are aimed not just at how Oklahoma can improve its standards, but also at how 
to make them competitive with the best state and international standards we have identified to 
date. 

Brief biographies of Achieve’s experts and consultants who participated in the standards 
benchmarking and assessment analysis for Oklahoma can be found in the Appendix. 
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Results for Oklahoma 



Major Findings: English Language Arts Standards Benchmarking 

The following summary highlights and explains the most important findings from Achieve ’s 
benchmarking study of Oklahoma’s K-12 standards in English language arts and their 
relationship to ACT’s Standards for Transition in English and reading. 

Strengths of Oklahoma ’s Priority Academic Student Skills (PASS) for English Language Arts 
and act's Standards for Transition in English and Reading 

1. Oklahoma’s standards in English language arts are comprehensive; essential 
knowledge and skills are present, and no significant gaps exist. 

Oklahoma’s standards generally include the knowledge and skills necessary for all students to 
achieve proficiency in reading and writing, and placement of content is, with few exceptions, 
grade appropriate. The breadth of coverage of the content described in PASS is quite close to 
that included in benchmark standards. These standards would provide a more-than-adequate 
foundation on which to build a successful curriculum for Oklahoma’s classrooms. Oklahoma’s 
treatment of research and visual literacy, for example, goes beyond the benchmarks at some 
grade levels. Reviewers noted that K-3 expectations, in particular, compare favorably to 
benchmark standards in terms of comprehensive coverage of essential areas in English language 
arts. 

2. Most Oklahoma English language arts standards are expressed in measurable terms. 

With a few notable exceptions, the standards emphasize performances that are measurable. For 
example, students are regularly asked to “identify,” “analyze,” “construct,” “evaluate” and 
“describe.” Oklahoma may wish to further strengthen this aspect of its standards by addressing 
those instances where measurability is a concern. In some instances, standards employ verbs that 
refer to learning activities, such as “participate” or “listen,” or to cognitive processes, such as 
reading expectations calling on students to preview, activate prior knowledge, monitor, cross- 
check, develop mental pictures and adjust their reading rate. These behaviors cannot be 
observed. What is verifiable is the ability of students to comprehend what they have listened to 
or read after using these and other strategies. This is not to say that reference to these strategies is 
not permissible in a standards document, but rather that the strategies should not be stated in the 
standard itself (many states allude to such internal processes in their documents, understanding 
that students equipped with these skills — measurable or not — tend to be good readers). The 
key to writing a measurable standard, however, is to focus on the results, rather than the 
processes of student learning. 

On occasion, there is a standard that will be hard to assess. For example, in Vocabulary sections, 
how will Oklahoma know when students “Begin to/Continue to” recognize? How will assessors 
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know if students have increased, extended or expanded their learning? Increased, extended or 
expanded from what? And by how much? 

Finally, with respect to measurability, some standards are challenging to assess due to their 
imprecision and level of generality. A notable example is in the grades 9-12 literature standard: 
“Demonstrate a knowledge of and an appreciation for various forms (genres) of literature, such 
as shiort story, novel, drama, narrative and lyric poetry, essay, and informational texts.” A lack of 
specificity or detail outlining increasing complexity through the grades makes it difficult to build 
a fair assessment. Assessment developers can — and often do — interpret the standards 
themselves and add the necessary precision by virtue of the kind of questions they ask on a test. 
The problem is students and teachers are not privy to those decisions until after the fact. Yet the 
whole purpose of standards is to set expectations up front and take the “gotcha” out of test 
taking. 

Reading Habits includes expectations that students select books and read daily. The emphasis is 
a good one, as we know that the more students read, the better readers they will become. The 
only addition that would strengthen this section would be to add minimum time limits or 
minimum amounts of daily, weekly, monthly or annual reading. 

3. The Oklahoma standards document is written for the most part in clear, jargon-free 
prose that is accessible to the public. 

Overall, the language of PASS is straightforward and understandable to a wide range of 
audiences, including students, parents, teachers and the general public. The document includes 
some terms, most referring to the process of learning, that are likely to be familiar only to 
educators. Phrases such as “balanced reading program,” “cueing,” “cooperative learning,” 
“Reader’s Theatre,” “recursive process,” “Rule of Thumb,” “Sustained Silent Reading/Drop 
Everything and Read” and “writer’s workshop” could be deleted, as they are not central to what 
students are expected to learn and be able to do. That said, Oklahoma was generally careful to 
place them in parentheses as samples and also to define them in the glossary. In the end, 
Oklahoma may decide that their value to teachers as concrete examples of learning strategies 
should prevail over the value of limiting the language of the standards to that immediately 
accessible to the public. 

4. Using asterisks in Oklahoma’s standards to flag content to be assessed at the local level 
is very helpful. 

The use of asterisks to distinguish standards that districts are to assess from those the state is to 
assess, while imperfectly applied, is a clearway to communicate who is responsible for what. 
More care should be taken to ensure that these signals are consistent across grade levels and do 
not overlook some elements that should be assessed at the state level. For example, “Contrast 
points of view (first and third person) in narrative text and explain how they affect the overall 
theme of work” or “Demonstrate a knowledge of literary elements and how they affect the 
development of a literary work” currently are marked with an asterisk, indicating they are not to 
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be assessed by the state. But failing to assess these and other such elements at the state level may 
weaken the power of the standards over time. Teachers will tend to teach what gets measured. 

Further confusion is caused by the fact that several standards that seem especially difficult to test 
at the state level were left without an asterisk. These are a few examples: 

• “Create mental images from picture and print.” 

• “Use a variety of electronic media (television, video, computer sources) to help them in 
developing their own stories.” 

• “Clarify meaning by rereading, questioning and checking whether text makes sense.” 

• “Use appropriate strategies for studying and learning content area text.” 

• “Identify and credit reference sources in a researched report.” 

• “Expand vocabulary through word study, literature and class discussion.” 

Moreover, certain content with an asterisk in one grade is without an asterisk in other grades. On 
occasion certain content that is duplicated within a grade has an asterisk in one part and is left 
without an asterisk in another part. For example, “Analyze the recurrence of archetypal 
characters, settings, and themes in world literature” is marked, yet “Recognize and discuss 
universal themes (archetypal patterns) in literature” is not. Both appear in the grades 9-12 
standards. 

5. Although Oklahoma’s standards in English language arts and ACT’S Standards for 
Transition in English and Reading have different emphases, reflecting their different 
purposes, the documents are fundamentally compatible. 

PASS is much broader than ACT’s Standards for Transition and is meant to describe a 
comprehensive set of outcomes that reflect a whole curriculum. This is an essential strength of 
Oklahoma’s standards. ACT’s standards, which are derived from the ACT test, are not designed 
to match a particular set of state standards, and the test assesses a nanower band of knowledge 
and skills. Not surprisingly, ACT assesses only a subset of the elements in Oklahoma’s 
standards. However, that subset is more specific and appears to be more rigorous than 
Oklahoma’s grades 9-12 standards. 

Rigor depends to some degree on how well progression is articulated across grade levels and also 
is a function of how precisely expectations are delineated. ACT’s Standards for Transition 
exhibit a clear progression of knowledge and skills across score ranges. At times, the quantity 
and sophistication of what is expected of students increases; at other times, it is the quality or 
complexity of the passages students are expected to read or work with that increases. 
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Oklahoma’s standards do not increase in intellectual demand consistently; distinctions between 
what is being asked at different grade levels often are blurred. 

A key difference in Oklahoma’s standards and ACT’s Standards for Transition is in the level of 
precision used to frame the two standards documents. The contrast in precision is readily 
apparent in the following examples: 

• Oklahoma’s standards expect students to determine the purpose of a reading passage. The 
ACT expects students to “Identify both the focus and purpose of a fairly involved essay, 
applying that knowledge to determine the rhetorical effect of a new or existing sentence, 
or the need to add supporting detail or delete plausible but irrelevant material.” 

• Oklahoma’s standards expect students to know how to use transitions. The ACT expects 
students to “Consider the need for introductory sentences or transitions, basing decisions 
on a complete understanding of both the logic and rhetorical effect of the paragraph and 
essay.” 

• Oklahoma’s standards expect students to write a variety of types of sentences, use 
parallel structure, conjunctions, etc. The ACT expects students to “Work comfortably 
with long sentences and complex clausal relationships within sentences, avoiding weak 
conjunctions between independent clauses and maintaining parallel structure between 
clauses.” 

• Oklahoma’s standards call for students to understand subject-verb agreement. The ACT 
expects students to “Ensure that a verb agrees with its subject when a phrase or clause 
between the two suggests a different number for the verb.” 

• Oklahoma’s standards expect students to identify supporting details. The ACT expects 
students to “Locate and interpret minor or subtly stated details in complex passages.” 

• Oklahoma’s standards expect students to make comparisons, conclusions and 
generalizations. The ACT expects the same but further tailors its standard to include 
generalizations, etc., “that reveal a feeling for the subtleties in relationships between 
people and ideas in virtually any passage.” 

• Oklahoma’s standards expect students to identify cause-effect relationships. The ACT 
expects students to “Identify implied, subtle or complex cause-effect relationships in 
virtually any passage.” 

The upcoming revision of Oklahoma’s standards is an opportunity for the state to consider 
adapting the language of ACT standards to fortify those Oklahoma standards and objectives 
flagged by Achieve reviewers as needing greater rigor and/or specificity. For example, when 
Oklahoma’s document speaks of subject-verb agreement, the more-specific ACT expectation 
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could be added underneath it. That way, the two documents could together forge a more 
comprehensive, coherent accountability system. 

Areas for Improvement 

Despite the strengths, reviewers found several areas that should be addressed for Oklahoma to 
use both the Oklahoma and ACT standards to substantially raise student achievement. 

Oklahoma’s standards sometimes lack specificity, which clouds the level of rigor 
expected. 

Striving for an appropriate level of specificity is worthwhile because specificity affects 
progression and rigor. As with all criteria for standards, there are trade-offs. When standards are 
too narrow, they appear to be part of a checklist, and the overall significance of a concept or skill 
may well be lost. On the other hand, when standards are broad or vague, they are vulnerable to 
wide differences in interpretation, and teachers, students and parents are left with insufficient 
direction. Widely varying interpretations may result in students experiencing very different 
treatments of essential content. 

Two factors determine the level of demand of an English language arts standard. One is the 
complexity and sophistication of the task, i.e., what students are asked to “do” with text. The 
other is the complexity and sophistication of the text itself that students are asked to read. 
Consequently, to make the expectation clear at a given grade level, it is important to identify not 
only the skill, but also the type of text to which the skill is applied. To illustrate, at grade 
under Responding to Text, one of the listed skills is “Draw conclusions, make predictions and 
comparisons based on what is read.” The demand of the standard means one thing when applied 
to Frog and Toad Are Friends and quite another when applied to The Trumpet of the Swan. 

Standard I.B.l is a case in point. It states that students “Read and comprehend both fiction and 
nonfiction that is appropriately designed for the second half of’ the grade level being discussed. 
Without further grade-level elaboration, one is hard pressed to determine progression or the level 
of rigor intended. A greater degree of clarity and specificity clearly is necessary if Oklahoma 
teachers are to come to consensus and bring a consistent focus and set of expectations to their 
classrooms. One approach is for Oklahoma to extend the standard to include a description of the 
kind of text anticipated in order to make the state’s expectation more concrete. Another is to 
offer a series of sample grade-level reading passages, as in Massachusetts’ standards, or a sample 
reading list, as in New Standards’. Either approach would greatly support a shared understanding 
of what a cognitively demanding and abstract skill, such as comprehension, means for a given 
grade level. 

Reviewers found that, in general, the Oklahoma standards seem to overspecify reading skills and 
underspecify writing skills, both in terms of the amount of attention Oklahoma gives to each area 
and in comparison with benchmark standards. As previously noted, the specificity of the early 
reading and mechanics skills focuses attention of both instruction and assessment on lists of 
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separate, enabling skills without ever drawing attention to the role these skills play in the broader 
context of literacy. At the same time, the areas of writing and listening and speaking provide 
inadequate guidance for instruction or assessment. The standards, for example, fail to capture the 
parallel, and mutually reinforcing, development of reading and writing skills through the grades 
and their relationship to the development of oral language skills, in spite of some reference to 
integration. Moreover, the discrete categories that are easily represented by itemized lists such as 
phonemic awareness, grammar/usage and mechanics receive a much greater, exhaustive, and 
sometimes redundant, treatment than do the more cognitively complex behaviors. The net result 
is something of a forest-and-trees problem: A sense of the direction and character of knowledge 
growth in the subject gets swamped by the grade-by-grade details that lack a context to make 
them meaningful. 

Oklahoma’s standards in English language arts lack a clear and consistent progression 
of knowledge and skills frOm grade level to grade level. 

While some Oklahoma standards increase in depth and sophistication, the wording of too many 
other standards is identical or nearly identical. For example, under the GrammarAJsage and 
Mechanics strand. Objective B reads exactly the same — “Identify and write the five parts of a 
letter with correct capitalization and punctuation” — for grades 3, 4 and 5. 

Another example is reading speed, where Oklahoma falls back on the same injunction to 
“increase reading speed” for grades 3 through 12: 

Grade 3: Increase reading speed through daily independent reading. 

Grade 4: Increase silent reading speed through daily independent reading. 

Grade 5: Increase silent reading speed through daily independent reading. 

Grades 6—8: Increase reading speed and comprehension through daily independent reading. 

Grades 9-12: Increase reading speed and comprehension through daily independent reading. 

No guidance is provided regarding what such an increase would resemble. Texas, however, 
provides the following information within its standards for grade 2, giving guidance beyond 
merely “read faster.” Its standard states, “Read regularly in instructional-level materials that are 
challenging but manageable (texts in which no more than approximately 1 in 10 words is 
difficult for the reader; a ‘typical’ 2"*^ grader reads approximately 70 wpm).” In contrast to 
Oklahoma standards in English language arts, repetitions and redundancies are minimal in the 
California and Massachusetts benchmark documents, and intellectual progression is much more 
clear as one moves up the grades. 

Sometimes there seems to be an inadvertent misplacement of standards when an earlier grade’s 
standard is drawn more broadly than a higher grade’s standard, thereby making it appear that the 
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earlier grade is, in fact, the more demanding. For instance, in grade 2 students are expected to 
write “ thank you notes and invitations,” while grade 1 expects students to write “thank you 
noiQS, friendly letters and invitations.” And at grade 2, students are expected to know how to use 
“commas” in general (implying all uses), while at grade 3 students are expected to know how to 
use commas in specific instances — “dates, addresses, series, quotes, and introductory words.” 

In both of these instances, the demands seem to lessen in the higher grades rather than increase. 

Another trouble spot is the concept of “fluency” in reading. It is fundamental but is left too vague 
in Oklahoma’s standards to describe increasing grade-level expectations. The overview of the 
PASS Language Arts section defines fluency as “achieving speed and accuracy in recognizing 
words.” If this term were accompanied by rate and accuracy measures directly linked to a set of 
agreed-upon reading levels, such as those described in the Reading Recovery program or by 
reading experts Irene Fountas and Gay Sue Pinhell, determining a progression of demands across 
the grades would be more straightforward. For example, Texas pins down its expectations for 
grade 1 students in the following way: 

1 .9 Reading/fluency. The student reads with fluency and understanding in texts at 

appropriate difficulty levels. 

The student is expected to: 

A. Read regularly in independent-level materials (text in which no more than 
approximately 1 in 20 words is difficult for the reader). 

B. Read regularly in instructional-level materials that are challenging but manageable 
(texts in which no more than approximately 1 in 10 words is difficult for the reader; a 
“typical” first grader reads approximately 60 wpm). 



A standard that offers a definition of reading speed and helps pinpoint the ability of an individual 
reader would go a long way toward bolstering effective reading instruction and guaranteeing a 
common understanding of “fluency.” 

Reviewers expressed concern that PASS standards do not reflect awareness of the parallel 
development of reading and writing skills through the grades and their mutual relationship to the 
development of oral language skills, in spite of some references to integration. Achieve’s 
experience has been that states find it very challenging to lay out a continuum of knowledge and 
skills in such a way that increasing intellectual demand and essential learnings are readily 
apparent. After states have done the hard work of defining content for grade levels, they 
sometimes omit the next, necessary step of checking content across grade levels, strand by 
strand, to ensure that expectations increase in complexity and that intermediate, enabling 
knowledge and skills have not been overlooked. Getting the right level of specificity in the 
standards is a key step in clarifying progression and rigor, and sharpening focus. 
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The present strands in English language arts overlap and do not present a coherent 
picture of the subject area. 

The number of strands in a subject area that a state chooses to include in its standards has a 
major effect on the overall clarity of the document. Oklahoma’s current sets of eight strands in 
grades 1 through 5 and 10 strands in grades 6 through 12 provide a confused picture of English 
language arts as either a skill area or a curriculum. As the standards are now organized, strands 
overlap in the areas of comprehension and literary analysis; this leads to repetition and 
redundancy. To illustrate, in grades 6 through 12, Strand I deals with Reading Processes, Strand 
II with Response to Text, Strand III with Information and Research, and Strand IV with 
Literature. All of these sections detail expectations about comprehending and interpreting both 
fiction and nonfiction materials, often repeating in one section what is already expressed in 
another. In some instances, the meaning of the strands seems to change across grade levels. 
Visual Literacy, for example, at some points seems to mean media study, at some points means 
use of word processing and the Web, and at some points is called Visual Literacy and Literature 
and includes reading of multicultural texts. 

Oklahoma may wish to adopt a simpler approach and combine Strands I-IV as subsections of 
Reading; Strands V and VI as subsections of Writing; keep Speaking and Listening as is; and 
recast Visual Literacy as either Media Study (video, film, television) or Use of New 
Technologies (word processing, the Web, database software). Another alternative is to drop 
Visual Literacy and redistribute the skills to other strands as part of Reading and Writing. 

To simplify the structure even more, creating standards that separate reading nonfiction from 
fiction could help. Such a structure could look like: 

Reading 

A. Reading Process (includes at the appropriate grade levels Print Awareness, 
Phonemic Awareness, Phonics, Vocabulary, Effective Reading Habits and 
Fluency) 

B. Comprehending Fiction 

C. Comprehending Nonfiction 

Writing 

A. Grammar/Usage/Mechanics 

B. Composing (includes writing for different audiences and purposes, variety of 
genres, writing process) 

Listening and Speaking 

Media Study 
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The standards should be more focused to signal the most essential content at each grade 
level. 

Focus is an evaluation of the explicit choices a state has made about the relative importance of 
any standard or set of objectives. Since Oklahoma gives all standards and objectives the same 
weight, it is not clear what the state regards as more or less important. All of the objectives read 
as equally significant; none are cast as stepping stones to achieving literacy in the broad sense of 
the term. 

State standards can express an implicit value by the sheer numbers of objectives or standards 
devoted to an area of English language arts. This tends to be the case with Oklahoma where an 
implicit value is suggested by the number of objectives assigned to discrete reading skills at the 
early grades and the large number of mechanics, grammar and usage objectives at all grade 
levels. Because these objectives are numerous, clear and explicit, they may inadvertently 
promote an unbalanced instructional program — one that favors phonics, grammar and 
punctuation over end products such as comprehension, composition and speaking. Said another 
way, phonics, grammar and punctuation are means to an end, not an end in themselves. That 
these particular skills also lend themselves to a checklist may send the message that effective 
instruction is achieved by addressing each of these elements as a separate entity, rather than as 
means to the end of competent reading and writing. Lengthy lists of writing qualities, for 
example, are not the way to sharpen focus. To the contrary, standards would convey a much 
stronger message regarding the balance of instruction if they were to address the characteristics 
of fluent, effective language and structural issues in writing, along with the expectation for 
competent editing. 

The New Standards benchmark document offers an alternative approach that is highly specific 
about both reading skills and grammar/usage but presents these skills in ways that keep the 
spotlight on the overall purpose of instruction. As an example, compare the standard from New 
Standards on kindergarten-level letter recognition to Oklahoma’s: 

New Standards Reading Standard 3: Children leaving kindergarten should know the letters of 
the alphabet and many of their corresponding sounds. The precise number of letters and sounds 
kindergarteners should know is not important; what is essential is that children grasp the idea of 
how letters represent sounds. We expect children leaving kindergarten to use their knowledge of 
sounds and letters to write phonetically, representing consonant sounds with single letters in 
correct sequence. 

Oklahoma III. The student will demonstrate the ability to apply sound-symbol 
relationships. 

A. Identify the alphabet by name (upper and lower case). 

B. Distinguish upper and lower case letters. 

C. Identify letters in first and'last name. 

D. Know sounds of the alphabet. 

E. Distinguish begirming and ending consonant sounds. 
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The New Standards expectations are much more clear about what is important for students to 
know and indicate how students should put their knowledge to use. 

The issue of focus in PASS for English language arts is complicated by the lack of consistent 
organization across grade levels as categories appear and disappear and standards are repeated 
across categories. Comprehension is the most confused area with related content distributed 
among the following categories: 



Grade 

1 


Reading 

Process 


Responding 
to Text 


Information 
and Research 


Effective 

Reading 

Habits 


Visual 

Literacy/ 

Literary 

Response 






Grade 

2 


Reading 

Process 


Responding 
to Text 


Information 
and Research 


Effective 

Reading 

Habits 


Visual 

Literacy/ 

Literary 

Response 






Grade 

3 


Reading 

Process 


Responding 
to Text 


Information 
and Research 


Effective 

Reading 

Habits 


Visual 

Literacy/ 

Literary 

Response 






Grade 

4 


Reading 

Process 


Responding 
to Text 


Information 
and Research 


Effective 

Reading 

Habits 




Visual 

Literacy 




Grade 

5 


Reading 

Process 


Responding 
to Text 


Information 
and Research 


Effective 

Reading 

Habits 




Visual 

Literacy 




Grades 

6-8 


Reading 

Process 


Responding 
to Text 


Information 
and Research 


Effective 

Reading 

Habits 




Visual 

Literacy 


Literature 


Grades 

9-12 


Reading 

Process 


Responding 
to Text 


Information 
and Research 


Effective 

Reading 

Habits 




Visual 

Literacy 


Literature 



Even within a grade, some standards are repeated. Some literature standards (within a single 
grade or grade span) fall under Comprehension, others under Responding to Text and still others 
under Literature. For example, at grades 9 through 12, theme, characteristics of different literary 
genres, figurative language, archetypal patterns, cultural diversity and strategies to comprehend 
written material are dealt with under at least two of these sections. Vocabulary and drawing 
conclusions — two other areas — are dealt with in two sections within the same grade as well 
(Vocabulary and Literature; and Literature, Information and Research). 

V The development of writing skills is not clearly articulated in Oklahoma’s standards. 

The treatment of writing is a major concern given that the current writing standard does not 
provide an adequate scaffold for developing student proficiency over time. The standard is 
divided into parts that divorce expectations for the use of the writing process from expectations 
for the types of writing. And while criteria for effective writing are included in elements in both 
of these sections, in neither section do they create a clear set of guidelines for effective writing in 
any one genre. Examination of the standards reveals an imbalance with many more demands 
included in the area of GrammarAJsage and Mechanics and many fewer in the area of Writing 



Measuring Up - Oklahoma 



30 



28 



Achieve, Inc., 2002 



than are present in the benchmark documents. New Standards and California, two of the 
benchmark standards, handle writing with much greater specificity and clarity. For example, 
focusing just on the early grades, one finds there are virtually no references to the quality of 
writing that Oklahoma expects of its students. New Standards, on the other hand, includes many 
statements, such as the following: 

• “Incorporate transitional words and phrases appropriate to thinking.” 

• “Embed phrases and modifiers that make their writing lively and graphic.” 

• “Develop a narrative or retelling containing two or more appropriately sequenced events 
that readers can reconstruct easily, which the author then often reacts to, comments on, 
evaluates, sums up or ties together.” 

• “Demonstrate a growing awareness of author’s craft by employing some writing 
strategies, such as using dialogue, transitions or time cue words; giving concrete details; 
and providing some sense of closure.” 

It is certainly true that from the early grades forward, students in Oklahoma are expected to write 
descriptive, narrative, expository and persuasive compositions; creative stories and poems; and 
letters, invitations and journals. However, unlike the New Standards standards, Oklahoma 
standards contain little or no description of the components or quality of writing expected. What 
does a middle school persuasive essay look like as opposed to One written in grade 5? What does 
a high school narrative look like as opposed to a middle school one? Grade 12 compositions are 
going to be more complete and of higher quality, yet the standards are virtually silent on this 
issue. Oklahoma standards force teachers and assessment developers to guess about the 
parameters. 

In terms of describing the qualities of effective writing within genres, California’s standards at 
grade 8 provide details that clearly describe the expectations for the grade level: 

2.0(W) Students write narrative, expository, persuasive, and descriptive essays of at least 
500 to 700 words in each genre. Student writing demonstrates a command of standard 
American English and the research, organizational, and drafting strategies outlined in 
Writing Standard 1.0. 

2. 1(W) Write biographies, autobiographies, short stories, or narratives: 

a. Relate a clear, coherent incident, event, or situation by using well-chosen details. 

b. Reveal the significance of, or the writer’s attitude about, the subject. 

c. Employ narrative and descriptive strategies (e.g., relevant dialogue, specific action, 
physical description, background description, comparison or contrast of characters). 
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2.2(W) Write responses to literature: 

a. Exhibit careful reading and insight in their interpretations. 

b. Connect the student’s own responses to the writer’s techniques and to specific 
textual references. 

c. Draw supported inferences about the effects of a literary work on its audience. 

d. Support judgments through references to the text, 'other works, other authors, or 
personal knowledge. 

2. 3 (W) Write research reports: 

a. Define a thesis. 

b. Record important ideas, concepts, and direct quotations from significant 
information sources and paraphrase and summarize all perspectives on the topic, 
as appropriate. 

c. Use a variety of primary and secondary sources and distinguish the nature and 
value of each. 

d. Organize and display information on charts, maps, and graphs. 

2.4(W) Write persuasive compositions: 

a. Include a well-defined thesis (i.e., one that makes a clear and knowledgeable 
judgment). 

b. Present detailed evidence, examples, and reasoning to support arguments, 
differentiating between facts and opinion. 

c. Provide details, reasons, and examples, arranging them effectively by anticipating 
and answering reader concerns and counterarguments. 

2.5(W) Write documents related to career development, including simple business letters 
and job applications: 

a. Present information purposefully and succinctly and meet the needs of the 
intended audience. 

b. Follow the conventional format for the type of document (e.g., letter of inquiry, 
memorandum). 

2.6(W) Write technical documents: 

a. Identify the sequence of activities needed to design a system, operate a tool, or 
explain the bylaws of an organization. 

b. Include all the factors and variables that need to be considered. 

c. Use formatting techniques (e.g., headings, differing fonts) to aid comprehension. 

In contrast, Oklahoma’s standards at grades 6-8 list only the kinds of writing required, noting 
little about the qualities of such genres. 
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Oklahoma, grades 6-8: 

V.B. Communicate through a variety of written forms and for various audiences and 
purposes (to inform, to persuade, to request, to argue, to entertain, to instruct, to 
describe). 

1 . Narrative, descriptive, expository, and persuasive paragraphs and longer compositions 
that establish and support a central idea with a topic sentence; supporting paragraphs with 
facts, details, explanations, or examples; and a concluding paragraph that summarizes the 
points. 

2. Creative stories, plays, and poems using figurative language such as alliteration, 
personification, simile, and metaphor. 

3. Friendly and business letters, “thank you” notes, and invitations. 

4. Editorials, reviews, and instructions. 

5. Journals. 

6. Reports. 

In fact, the grades 9-12 Writing standard includes virtually the same objective as number 1 
above at grades 6-8, changing only the demand from “paragraphs” to “compositions” and 
substituting “thesis statement” for “topic sentence.” Benchmark standards all expect that grade 8 
students are able to produce multiparagraph writing, yet Oklahoma’s standards expect merely 
paragraph length by the end of grade 8. While the expectation that genres should also include 
“reflective writing” seems to be an attempt to increase the demand of this objective, there is no 
definition of the term even provided in the glossary. 

It also is important to note that gaps in the writing standards at the high school levels, 
particularly in the areas of organization, word choice and stylistic elements, cause a mismatch 
with act’s English Standards for Transition, which focus heavily on these aspects of 
composition. 

Major Findings: English Language Arts Alignment of Tests to 
Standards 

Oklahoma’s Core Curriculum Tests in English language arts for grades 5 and 8 each consist of 
50 multiple-choice items. Five items in each test are designated as field-test items. The English II 
test consists of 100 items, 20 of which are designated as field-test items. These tests are not 
strictly timed; sessions can be extended up to one hour for students who require additional time. 

A separate, on-demand writing test is currently administered at grades 5 and 8; a corresponding 
high school writing assessment, being field tested this year, is scheduled for implementation in 
school year 2002-03. 
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The ACT English test consists of 75 multiple-choice items and does not include an on-demand 
writing component. The ACT reading test consists of four 750-word passages taken from 
published sources with 10 multiple-choice items per passage. Both tests are strictly timed in 
standard testing conditions. Although the ACT is mapped by the company to its own Standards 
for Transition, Achieve was asked to map ACT items to Oklahoma’s standards to determine how 
closely PASS and higher education’s expectations for college success align. In this way, at the 
close of the analysis, the reviewers would be able to comment on whether a curriculum based on 
PASS would support students’ success on the ACT. The following summary highlights and 
explains the most important findings from the study of the alignment of Oklahoma Core 
Curriculum Tests and ACT assessment to Oklahoma standards (Achieve examined only Form 
57B of the ACT 1997-98 test). More detailed findings are included in the accompanying secure 
technical report. 

Strengths of the Assessments 

Oklahoma’s Core Curriculum Tests in English language arts are strongly aligned to 

those standards identified as being appropriate for testing at the state level. 

Oklahoma clearly signals which standards are “fair game” for state testing and which should be 
tested at the local level, both in its standards document and in the accompanying test 
specifications. The test specifications, however, using the English II test as an example, identify 
only 63 percent of the standards and a much lower percentage at the more specific subcategory 
or objective level. This low percentage of items addressed comes about for two reasons: Many of 
the standards and subcategories are not appropriate for assessment on a large-scale test; and, as 
was noted earlier in the standards review, Oklahoma’s standards in English language arts often 
are redundant. 

In terms of content tested as compared to content contained in the standards, reviewers 
determined that 71 percent of items on the grade 5 test, 80 percent of items on the grade 8 test 
and 75 percent of items on the English II test are strongly aligned. Nearly all of the remaining 
items were found to be at least partially aligned, but the overly general nature of some of the 
standards kept reviewers from being sure of an exact match. 

However, when reviewers compared the performances required by items to those described in 
the standards, the match between test items and standards was somewhat less strorig than the 
match found for content (62 percent of items at grade 5, 73 percent of items at grade 8, 56 
percent of reading items and 100 percent of English II language arts items are in alignment with 
the standards). With few exceptions, the remaining items are partially aligned to the standards in 
terms of performance. Instances of partial alignment stem from the limitations of a multiple- 
choice format in assessing certain kinds of performances, as well as from the overly general 
descriptions of some performances contained in the standards. 
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In summary, while the Oklahoma tests are more closely aligned to the content described in the 
standards than to the performances indicated, all three tests assess the knowledge and skills in the 
standards earmarked for state, as opposed to local, assessment. 

Reading passages on the Core Curriculum Tests are well chosen; they are grade- 
appropriate, engaging and representative of the variety of reading expected by the 
standards. 

Reviewers indicated that the selection of reading passages is a praiseworthy feature of all three 
tests. This stands in contrast to a number of other state tests reviewed by Achieve in which the 
choice of passages was disappointing. Passages on Oklahoma’s tests are keyed to the respective 
grade levels, hold the reader’s interest, and represent an excellent balance of informational text 
and fiction. For example, the nonfiction pieces on the grade 8 tests are well constructed and well 
assessed. Similarly, the passages on the English II test represent both a range of topics and time 
periods — from Longfellow to James Michener and from relatively current events to historical 
ones. With respect to the 8‘^-grade test, reviewers noted that the fiction pieces were not excerpted 
from longer pieces as skillfully as they could be and that the passages taken as a whole do not 
offer enough challenge for high-performing readers — all the text was straightforward with little 
nuance of language or complexity of structure. 

Oklahoma’s decision to include on-demand writing tests as part of its assessment 
system is a good one. The state is well positioned to strengthen further the writing 
component by working with higher education to ensure a consistent level of rigor. 

Reviewers applauded the inclusion of on-demand writing in Oklahoma’s Core Curriculum Tests. 
Without question some aspects of editing skills, for example, can be reasonably well assessed 
with multiple-choice items. That said, it is very difficult to determine how well students can 
respond to the totality of a writing task — identifying the audience and purpose; organizing the 
piece; and attending to grammar, syntax and punctuation — without asking them to actually 
compose on the spot. However, getting a realistic picture of a student’s writing proficiency is but 
one benefit of direct writing assessments; even more important is the positive effect on 
classroom practice that evolves in response to direct assessment. Teachers teach and students 
prepare for tests very differently when the format is multiple-choice as opposed to open-ended or 
essay questions. Selecting the best answer from a series of four possible responses is far less 
intellectually demanding than having to assemble one’s thoughts and package them persuasively. 
But the latter skill is the one required for students to succeed in knowledge-based careers and 
postsecondary schooling. Oklahoma should hold fast to its direct writing assessment and take the 
next step, linking student performance to job applications and college placement tests. By the 
same token, reviewers found the lack of a direct writing assessment to be a weakness of the ACT 
— one they hope will be addressed as online assessment becomes commonplace. 
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Areas for Improvement 

ACT assessments in reading and English are not well aligned to Oklahoma standards. 

Simply put, the ACT emphasizes more specific knowledge and skills than do the Oklahoma’s 
standards — an unsurprising finding given that Oklahoma’s standards are meant to frame a K-12 
curriculum while the ACT is mainly meant to assess readiness for academic success in college 
courses. It also is the case that ACT often addresses concepts more sophisticated than those 
addressed in the Oklahoma test. The ACT reading section, for example, requires students to 
make connections among pieces of evidence presented in the text and draw valid conclusions 
from them. ACT includes six items assessing such cause-effect relationships (PASS Standard 
II.H). Oklahoma had none. 

Reviewers also noticed a discrepancy, which did not arise in the protocol-based analysis of the 
ACT assessment as compared to Oklahoma’s standards, but rather surfaced in an informal 
comparison of reading passages on the ACT test and Oklahoma’s English II test. Rigor in 
assessing reading is as much a function of the text selected as the types of questions posed. Both 
tests include passages that are engaging, accessible and drawn from a variety of genres. But after 
correcting for the fact that the ACT is pitched to late high school and Oklahoma’s English II test 
to early high school, reviewers concluded that the ACT selections are more cognitively 
demanding in terms of language, structure, tone and topic. 

The reading section of the ACT provides four passages, each representing one genre: prose 
fiction, social science, humanities and natural science. One obvious reason for the lack of 
alignment in reading between the ACT and Oklahoma’s standards is the fact that the ACT does 
not assess literature in the way one would expect literature to be assessed from reading 
Oklahoma’s standards. While ACT includes a prose fiction selection, related questions are not 
directed at knowledge of literary elements or figurative language. Instead, questions assess 
whether the test taker understands the author’s meaning. In fact, reviewers therefore mapped 
these items to Oklahoma’s standard for Reading, rather than to Oklahoma’s standard for 
Literature. (By way of contrast, reviewers mapped 40 percent of Oklahoma’s English II test 
items in Reading to the Literature standard.) 

A possible consequence of the differences in level of text employed by the ACT and Oklahoma 
is that students who score well on Oklahoma’s English II test might find themselves struggling to 
achieve a comparable score on the ACT; they may not be prepared sufficiently for the 
complexity of the text or the subtlety of the accompanying questions they are likely to encounter 
on the ACT. 

Adjusting the reading passages on the English II test to include at least one “stretch” passage — 
aimed at high-performing 10‘^ graders and accompanied by questions that assess the more 
intellectually demanding parts of the related standards — would bring Oklahoma’s expectations 
closer to the performance levels required by the ACT. 
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The English section of the ACT measures knowledge of conventions and such analytic aspects of 
writing as topic development, organization, coherence, word choice and style. Since these 
aspects are not addressed specifically in Oklahoma’s standards, or assessed in its test, a gap in 
alignment is inevitable. Reviewers reported they were unable to match 35 ACT English items to 
Oklahoma standards. The following excerpts from Oklahoma’s standards, which address editing, 
and an example drawn from ACT’s released items underscore the kind of differences reviewers ' 
discovered. Oklahoma refers to the editing process in two of its writing objectives. 

V. Writing (PASS) 



Use a writing process to develop and refine composition skills. Students are expected 
to use prewriting strategies, write and revise multiple drafts, edit, and share their 
compositions. 

Select and use reference materials and resources as needed for writing, revising, and 
editing final drafts. 



When reviewers analyzed Oklahoma’s English II test they found that all of the Oklahoma editing 
items address sentence-level_e6\iu\^ issues — “What is the best way to rewrite the sentences in 
lines XX.” In contrast, ACT items often address paragraph-level edits or the even more 
demanding revision. Students may be required to resequence the order of 

sentences in a paragraph to strengthen the logical structure or to choose an option for change that 
would reinforce or alter the point of view in an entire piece. In the item below (from ACT’s 
released items) the student is asked to make a decision at a high level of revision, one that 
requires the student to consider organizational choices as means of conveying meaning and 
intent: 



The writer intends to add the following sentence to the essay in order to provide a 
comparison that would help underline challenges that Bessie Coleman faced: 

Her dream of becoming the world’s first black woman pilot seemed as 
remote in Chicago as it had been in Oklahoma. 

In order to accomplish this purpose, it would be most logical and appropriate to 
place this sentence after the: 

A. first sentence in Paragraph 2 

B. first sentence in Paragraph 3 

C. last sentence in Paragraph 3 

D. first sentence in Paragraph 5 

These kinds of items, and others like them, address issues of topic development in the context of 
the author’s purpose and focus on larger and more abstract issues than are addressed in 
Oklahoma’s editing items. 

The treatment of punctuation also is telling. The Oklahoma test includes four items, while ACT 
includes 16. In this case it is not merely the number of items that is an issue, but the skills 
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addressed. While the ACT test requires students to demonstrate an understanding of the use of 
colons in compound sentences, the semicolon, the dash and numerous comma rules, Oklahoma s 
items focus mainly on simple commas, devoting one item to capitalizing a proper noun. As was 
the case with reading, in classrooms that base their instruction solely on PASS, students may not 
be held to the high level of writing skills demanded by the ACT test. 

The state should consider incorporating the more demanding and varied skills measured by the 
ACT into PASS, while at the , same time avoiding elongated lists of editing topics. To inform 
teachers and students of the higher-level expectations in editing in a manner specific enough to 
guide instruction, perhaps PASS could be restructured so that only those skills that need to be 
introduced at each succeeding grade level are specified, with the understanding that the skills 
specified in the previous levels are to be expected as well. It is essential that students master 
those language skills — beyond simple capitalization and comma use — that are required to be 
college-ready and function in an information-based society. Oklahoma’s extension of its direct 
assessment of writing to the 10* grade also presents an opportunity to address the shortcomings 
identified by reviewers. 
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Major Findings: Mathematics Standards Benchmarking 



Achieve’s choice of benchmarks — Arizona’s standards and especially Japan’s and Achieve’s 
MAP standards (limited to the middle grades) — should be placed in the larger context of 
mathematics education reform across the states. Both Japan and Achieve’s MAP standards look 
to the future; no state, not even Arizona, has yet articulated standards of equal rigor, and no state 
is yet ready to implement and hold students accountable for what is admittedly a high level of 
achievement. Achieve’s benchmarking process is designed to compare a state’s standards to 
some of the best in the world. It is therefore unrealistic to expect a state’s standards, its teachers 
and its students to align with these external benchmarks overnight. The message to states has 
been and remains one of continuous improvement. 

The following summary highlights and explains the most important findings from Achieve’s 
benchmarking study of Oklahoma’s K- 12 standards in math and their relationship to ACT’s 
Standards for Transition in math. 

Strengths of Oklahoma ’5 Priority Academic Student Skills (PASS) for Mathematics 

1. Oklahoma’s PASS for mathematics in grades K-5 lays a strong conceptual foundation 
for further study and compares favorably with the benchmark standards in this regard. 

A particular strength of the math standards is that they lay a strong conceptual foundation, 
especially in the early grades; build upon that foundation in developing skill proficiency and 
fluency in a later grade (usually the following grade); and then expect students to abstract and 
apply those ideas as they are introduced to more sophisticated thinking. 

In addition, Oklahoma generally has made very defensible choices as to the grade-level 
placement of its skill expectations. The grade level at which PASS expects computational skills 
usually matches Arizona’s benchmark standards, although it sometimes is a year behind Japan’s 
standards. For example, Oklahoma 3'^‘*-grade students are expected to “develop fluency with 
single-digit multiplication facts and their related division facts.” Similarly, Arizona 3'‘*-grade 
students are expected to “demonstrate proficiency with basic facts up to the fives.” Japan, 
however, expects students in 2"‘* grade “to know about the multiplication table and to be able to 
correctly multiply 1 -digit numbers” and to extend this knowledge to two-digit numbers and to 
multiples of 10 in the 3'^'* grade. These differences with grade- level placement relative to 
Japanese standards were not of overriding concern to reviewers because of Oklahoma’s careful 
attention to providing a strong base for further development of students’ conceptual 
understanding. 

2. Overall, PASS for mathematics contains a thoughtful progression and well-developed 
sequence of knowledge and skills. 

Oklahoma’s explicit listing o( major concepts and maintenance concepts for each set of grade- 
level and course-level standards signals that students attain proficiency in math over time. 
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continually building on previous learning from one year to the next. The lists underscore the 
necessity of students’ retaining and augmenting their knowledge base by highlighting the 
essential prior knowledge students will be called upon to use in a particular grade as they 
encounter new concepts. Another positive aspect of PASS is that the overviews provided for 
kindergarten and grades 1-5, 6-8 and 9-12 do a good job of setting the philosophical context for 
standards and addressing Oklahoma’s assumptions about classroom materials, such as 
manipulatives and technology. Oklahoma also is clear about its intent to develop student 
understanding. It deliberately moves from the concrete to the pictorial to the abstract, providing 
the reader with important insight into the way the standards were developed. 

PASS for math succeeds in striking a generally good balance among the conceptual, procedural 
and application aspects of math. For example, as in Japan, kindergarten is seen as a time to 
explore the world from many points of view, two of which are numerical and spatial. The 
kindergarten math standards set a vision of students examining relationships among objects, 
exploring the results of those relationships and learning to describe both. The standards for the 
elementary grades orchestrate the development of numerical and arithmetic conceptual 
understanding, processes and skills. In and 2"'^ grades, students are engaged in the intentional 
exploration of quantitative and spatial aspects of the world. However, they also are beginning to 
learn to connect those explorations with such constructs as time, money, congruent shapes and 
patterns with symbols and to include operational components such as the addition and 
subtraction facts to 10, growing patterns, and estimation. Care is taken to first lay a foundation of 
understanding before diving into the development of fluency or proficiency of skills. Except for 
stretching computation and linear algebra across too many grades, Oklahoma content skills lay 
out a logical developmental sequence. 

That said, Oklahoma is less successful in establishing a clear progression of process skills. For 
example, process standards for grades 9-12 are not as explicit as those for grades 6-8, leaving 
the impression that they are not as demanding, even though some of the particulars are in fact 
more complex. The emphasis on multistep and nonroutine problems found in grades 6-8 is 
nowhere to be found in the process standards for grades 9-12. Similarly, it is not at all clear how 
the grades 9-12 communication expectations are more advanced or sophisticated than the grades 
6-8 expectations. Part of the difficulty stems from the fact that process skills are expressed 
inconsistently in different grade bands. They appear to have been written by different educators, 
who may not have fully coordinated their work. To realize a clear progression of skills, the 
expectations for the three grade bands (grades 1-5, 6-8 and 9-12) must build on one another, 
and the language used across the grades must make the linkages transparent. 

3. PASS for mathematics generally merits high marks for clarity, specificity and 
measurability. 

Overall, reviewers found PASS for math to be clear, specific and measurable. The organizational 
structure and format of the document is reader-friendly and useful. In addition, both content and 
process standards tend to be written for public consumption and are presented in simple, jargon- 
free language. 
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Whenever vocabulary is used in PASS for math that is judged to be unfamiliar to a general, 
reasonably well-educated audience, the word(s) used are written in boldfaced type in the body of 
the document and then included in a glossary appended to the standards document. Such a 
glossary can be helpful, but it would be a good idea to review the definitions and the terms 
included when revisions are next made to the standards themselves. It is not always clear why 
some terms are included and not others — for example, why “manipulatives” but not “models”? 
Also, some words, such as “explore” or “estimate,” may not need to be included in the glossary 
since ordinary people would likely understand what they mean. Finally, because of the need for 
conciseness, a number of mathematical terms (e.g., “congruent”) are defined in ways that are 
technically incorrect or circular. 

There are two kinds of jargon common to documents such as this — pedagogical (e.g., 
“manipulatives,” “represent,” “explore”) and mathematical (e.g., “slope,” “array,” “congruent”). 
The latter are inevitable but may not be accessible to the public as many parents either have not 
studied or have long since forgotten the meaning of such terms. Pedagogical jargon, however, is 
another matter. While the use of pedagogical jargon in Oklahoma’s documents is not particularly 
problematic, there are strategies the state can employ to make such language clear to parents and 
members of the community. For example, the term “process standards” has meaning to many 
math educators, particularly those familiar with the work of the National Council of Teachers of 
Mathematics (NCTM). To math educators, “process standards” are analogous to problem 
solving, communication, reasoning, connections and representation — words that any adult 
would know, but with special meanings that have developed within the community of math 
educators. Particularly when process standards are separated from content standards, as they are 
in Oklahoma’s PASS for math, it is very easy for teachers and test developers to lose sight of 
them. Oklahoma could take some very explicit steps to help clarify and make more concrete 
what teaching and learning of the process standards look like. This could be done most 
succinctly in the form of sample problems and assessment tasks. 

The numbering/coding system used to identify the content and process skills in the standards 
works fine, provided the reader is looking at the standards for only one grade. However, if the 
reader is looking across grades, the coding system does not provide adequate support for easy 
referencing or referral. At grades K-5, the Roman numeral designates a consistent standard 
throughout. For example. Patterns — analogous to an algebra standard — is/listed as Roman I, 
Number Sense as Roman II, and Number Operations and Computations as Roman III. This is not 
the case in grades 6-8, where standards are divided differently from grade to grade so that 
Roman II is Number Sense and Computation in grade 7 but is Algebraic Operations in grade 8. 
This causes confusion among grade-level divisions. Because the high school standards are 
arranged by course, the order and standards are changed again. This lack of consistency makes 
tracking ideas difficult and citing standards — by any means short of stating the standard — 
somewhat problematic, or at least not very useful. 

PASS provides sufficient guidance for curriculum planning for teachers who are well prepared in 
math. The document provides a clear picture of what students in each grade level (K-8) and each 
course (Algebra I, Geometry and Algebra II) are expected to know and be able to do. Suffice it to 
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say that not all teachers are adequately prepared in math and will require extensive support and 
professional development to translate these expectations into an effective, functioning 
curriculum. Oklahoma should stay the course with its plan to develop aligned curriculum 
materials and make these available to teachers electronically. 

Compared to the benchmark standards, Oklahoma’s math standards are stated in more specific 
language than Japanese standards while maintaining at least some of the conceptual foundation 
so valuable in the Japanese document. PASS is not as specific as Arizona’s benchmark standards 
but still manages to clearly convey the level of expectations so that they serve as a good map for 
assessment developers, teachers, parents and other education stakeholders. 

Most of Oklahoma’s math standards are expressed in measurable terms. For example, students 
may be asked to “multiply and divide,” “estimate,” “collect, organize, and interpret,” and 
“solve.” On occasion, however, there are standards that may be difficult to assess on a large- 
scale assessment. Oklahoma identifies these content skills and process skills with asterisks (*), 
indicating that they are to be assessed at the local school district level. The number of asterisked 
skills is held to a minimum, however, and includes expectations such as “build models,” “use 
geometric tools,” and “use technology to interpret and graph.” Oklahoma is to be commended for 
ensuring that their standards avoid the inclusion of unnecessary, nonacademic expectations. 

4. ACT’S Standards for Transition focus attention on essential content and skills in 
mathematics necessary for college admission. 

The Standards for Transition represent content knowledge that is reasonable to expect from high 
school students who are at least contemplating college attendance, and Oklahoma has made 
significant progress by using them to telegraph this information to teachers and students. 
Differences exist, however, between the Standards for Transition and Oklahoma’s PASS for 
Algebra I, Geometry and Algebra II. Care needs to be taken to ensure that teachers and students 
are aware of these differences. For example, the Standards for Transition tend to emphasize 
different aspects of geometry — and present geometry in a different way — when compared 
with PASS. Oklahoma’s PASS for Algebra I, Geometry and Algebra II puts all expectations 
related to Geometry and Measurement in one of two places — in the Geometry/Measurement 
strand of Algebra I or within Geometry. There are no expectations related to Geometry/ 
Measurement in Algebra II. The Standards for Transition, on the other hand, integrate geometric 
expectations into several content strands. In particular, expectations about coordinate geometry 
are in the Graphical Representations strand; those emphasizing such constructs as perimeter, area 
and volume are in Measurement; those involving properties of angles and special triangles are in 
Properties of Plane Figures; and those involving trigonometry are in Functions. The emphasis 
given by PASS and the Standards for Transition to particular topics also varies at times. For 
example, PASS mentions trigonometry once — as one of the Geometry content skills. The 
Standards for Transition, on the other hand, mention trigonometric concepts in five of the nine 
expectations specified for Functions. 
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As has already been noted, one of the strengths of Oklahoma’s math standards is their strong 
conceptual underpinning. While Oklahoma’s standards reflect a good balance between 
procedural, conceptual and application, the Standards for Transition tend to emphasize the 
procedural with an emphasis on skills (e.g., multiplying, writing, computing and solving). Verbs 
such as “interpret” or “analyze” are invoked only a limited number of times, generally at the 
upper two score ranges of the Standards for Transition. 

The Standards for Transition exhibit a clear progression of knowledge and skills across the score 
ranges, successfully teasing out incremental levels of content and complexity across score 
ranges. A good example is the Number domain, where it is important for students’ development 
— and for their continued work in college — that their understanding about numbers and data 
continue to mature. Another example is the Equations and Inequality standard, where the 
Standards for Transition are similarly effective in delineating increasing levels of demand across 
score ranges. As a student’s score on the ACT assessment rises, so does the likelihood that he is 
able to solve increasingly complex problems — as evidenced by the progression in the Standards 
for Transition from solving one-step equations with simple answers, to solving first-degree 
equations, to solving first-degree inequalities and simple quadratics, and, finally, to solving 
absolute value equations and inequalities. 

The progression in PASS is generally compatible with the progression evident in the Standards 
for Transition. Generally, the content in the first two score ranges (16-19 and 20-23) of the 
Standards for Transition is comparable to PASS expectations prior to high school. The Standards 
for Transition align with the 24-27 score range consist of expectations from both the middle 
school level and the Algebra I and Geometry course standards. Several items indicated by an 
asterisk or found in the highest score range (33-36) are found only in PASS Algebra II 
standards. 

Oklahoma students who successfully complete and learn the content and process skills indicated 
in PASS for grades K-8, Algebra I and Geometry will have the content knowledge they need to 
score at least in the 24-27 range, and likely in the 28-32 range, on the ACT. Complex numbers 
is the only topic in the 24-27 score range that may prove unfamiliar to students, while composite 
functions, special triangles and solving absolute value equations may be unfamiliar topics in the 
28-32 score range. To score in the highest range, Oklahoma students would be well served to 
have learned the content specified in Oklahoma’s Algebra II course. 

The Standards for Transition serve as a reasonably comprehensive measure of middle and high 
school mathematical content. Standards that address the areas of number and operations, 
probability and statistics, algebra and functions, geometry, and measurement all are there. 
However, the Standards for Transition are specific in their statement — and of a finer grain size 
than PASS — so they address certain aspects of each of these areas and not others. They appear 
to describe a subset of what one would want students to know and be able to do at a particular 
score range. When all score ranges are included, the comprehensive nature of the Standards for 
Transition is more apparent. However, while the Standards for Transition set a clear vision of 
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content expectations, they are not intended to be the foundation for a rich curriculum that would 
help students learn to think about and internalize the knowledge they have encountered. 

Areas for Improvement 

Oklahoma needs to clarify the level of rigor it expects in its standards for mathematics. 

The content and process skills delineated in Oklahoma’s PASS standards generally convey the 
breadth of mathematical content characteristic of solid K-12 programs. However, the content 
and process skills are less successful in conveying the depth of knowledge and skills required. 
Such a simple matter as requiring students to change the coefficients of a linear equation from 
integers to fractions to decimals, for example, can push students to deeper understanding, even 
though these are all instances of the same expectation. 

While reviewers applauded Oklahoma for ensuring generally good balance among the 
conceptual, procedural and application aspects of math, they did express concern that the 
standards, in some instances, stretch topics out too extensively from grade to grade. For example, 
the treatment of fractions and division spans across four or five grades, contributing to the 
criticism that too much arithmetic spills over from grades 1 through 5 into grades 6 through 8. 
Similarly, the topic of linear equations is revisited in three or four grade levels with the result 
that algebra is developed much more slowly and less completely than in the benchmark 
documents. In contrast, by the end of 8* grade, Arizona, Japan and MAP all have set more 
comprehensive and rigorous expectations relative to linear equations and delved more intensely 
into nonlinear functions than has Oklahoma. (PASS does not explicitly mention quadratic 
equations until Algebra II.) 

The danger in prolonging the teaching of arithmetic skills and linear equations is that Oklahoma 
will fall into the trap of having its math curriculum be a “mile wide and inch deep” — a general 
characteristic of U.S. math curricula criticized in the Third International Mathematics and 
Science Study (TIMSS). 

Ambiguity about depth is even greater when one considers the process skills. Oklahoma uses the 
“THINK STAR*” image (with content at its center and the five process standards at each point) 
throughout the document to reinforce and remind users that the math curriculum should integrate 
the process skills into content-centered lessons. While reviewers concur that it is important to 
help teachers integrate process expectations into content as teachers teach lessons, they question 
the effectiveness of this tool as the sole means of dealing with this issue. One reviewer expressed 
concern that such an icon may avoid the hard work of illustrating just how this can be done 
effectively and may provide an excuse for not developing the process standards in a grade- 
appropriate manner through concrete examples and sample problems tied to grade-specific 
content. What, for example, does the grade 68 process skill really mean when it says to “apply a 
variety of strategies to solve problems, with emphasis on multi-step and non-routine problems?” 
While this process skill offers some parenthetical expknation as to what this might include — 
restating the problem, looking for a pattern, diagrams, solving a simpler problem, working 
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backwards, trial and error — these strategies could be similar to those used by children in grades 



In order to leave no doubt about the level of rigor expected, some states and MAP include 
illustrative problems in standards. As it now stands, it is impossible to infer from PASS the level 
of cognitive complexity of problems that students are expected to be able to solve. For any given 
topic, the depth, complexity and connectivity of tasks that students should be able to carry out 
could range from trivial to levels that would stump most college graduates. In the next revision 
of its standards, Oklahoma may want to include sample tasks and concrete examples to peg the 
level of rigor expected. The lack of these anchors leaves the Oklahoma standards essentially 
silent regarding the intellectual level that may be required on its assessments. 

At the secondary level, course standards (especially those for Algebra I and Geometry) tend to 
describe a minimal set of expectations. It will be important that these skills be exceeded in strong 
algebra and geometry programs. It is most certainly the case that a stronger program than the one 
described by the standards will need to be in place to meet the needs of academically able 
students, especially those headed for careers in math, science and engineering. 

PASS standards for mathematics, while generally comprehensive, treat certain topics 

inadequately, particularly in algebra, probability, statistics and data analysis. 

Generally, the key aspects of mathematical thinking, mathematical process and mathematical 
skills that we would hope all of our high school graduates would learn are present in PASS. That 
said, reviewers voiced concern about the treatment of certain fundamental topics. In algebra, to 
reiterate, the standard’s almost singular focus on linear functions and equations is a comparative 
weakness relative to benchmark documents. It is important that Oklahoma high school graduates 
also understand the basic features of common nonlinear functions such as exponential growth 
and the normal curve. Exponential growth is critical in imderstanding such everyday phenomena 
as compound interest, return on investment and inflation. The normal or bell-shaped curve is 
important in understanding measurement and statistical applications in a variety of arenas — 
medical, social and industrial. Unfortunately, Oklahoma delays exponential and quadratic 
functions and equations until Algebra II, which may not be taken by all Oklahoma students. This 
will have consequences for the Geometry course in that it limits what can be done with circles 
and coordinate geometry. A second area where this gap will have effect is in delaying the 
discussion of physical phenomena, such as the motion of an object under the force of gravity. 
Oklahoma should examine closely the content of both its math and science courses to ensure that 
postponement of any discussion of nonlinear functions and equations, including quadratics, will 
not have a negative impact upon the intended science curriculum. 

In contrast, Japan expects all students to be able to understand and solve quadratic equations in 
grade 9. Arizona expects that by the end of grade 8, students will be able to “distinguish between 
linear and nonlinear functions, given graphic examples.” A core expectation of Arizona’s 
standards for grades 9—12 that is included on the state’s assessment includes solving quadratic 
equations having integral roots. Achieve’s MAP standards are even more rigorous and more 



Measuring Up - Oklahoma 



45 



Achieve, Inc., 2002 



specific, with their expectation that students exiting grade 8 be able to recognize simple 
nonlinear functions that arise in problem contexts and represent them using tables, graphs and 
formulas; transform and graph quadratic functions by factoring and completing the square; work 
fluently with common nonlinear functions that represent relationships such as the volume and 
radius of a sphere or the value of a bank deposit — the interest rate, compounding period and 
time lapsed; relate the solutions of a quadratic equation to the graph of the corresponding 
quadratic function; and solve a quadratic equation by factoring, finding where the graph of the 
fiinction meets the x-axis and completing the square. Students in Japan and Arizona would be 
significantly more advanced in algebra than their counterparts in Oklahoma by the end of grade 
10 given the expectations set for them concerning quadratic and other nonlinear functions and 
equations. 

Another concern is the omission of irrational numbers. While brief mention of “simplifying and 
evaluating expressions including; radical” is made in Algebra I standards, it is not clear that 
students will understand the idea of an irrational number. Clearly, if students are to determine the 
distance between two points on the coordinate plane, as is expected in PASS Geometry 
standards, they will need to know how to deal with square roots. 

As Oklahoma revises PASS for math, the state also will want to thoroughly review the level of 
expectation and the sequencing of content skills in probability, statistics and data analysis. 
Oklahoma’s treatment of this area is uneven across standards. While the foundational concepts 
appear to have been laid in the areas of statistics and counting (classical probability), students are 
not required to reach very far into the field. The statistical aspects of this standard, including the 
design of experiments, understanding of a random variable and working with the normal curve, 
should be developed more strongly. In addition, treatment of this standard at the middle school 
level appears superficial and less demanding than the expectations detailed for lower grades. 
Arizona clearly has more advanced expectations than Oklahoma does through grade 1 0, although 
Oklahoma has a particularly strong Data Analysis, Statistics and Probability standard in its 
Algebra II course. However, the placement of probability, statistics and data analysis 
expectations at the end of the three high school course sequence might be problematic, since the 
concepts might never get taught or receive the emphasis they deserve. One suggestion is to focus 
more of the middle grades expectations on a mix of algebra, geometry and 
probability/statistics/data analysis, giving the lower grades more of an opportunity to focus on 
and finish skills related to arithmetic and computation. The demands placed on the average 
citizen in our media-rich society today require a level of statistical and analytic skill not needed 
in earlier generations. 

Although PASS and ACT’s Standards for Transition have many similarities, a number 
of differences will need to be addressed if a standards-based system including these two 
components is to appear seamless. 

On the one hand, discrepancies between the two systems could be ignored since students who 
learn the content described in Oklahoma’s course standards for Algebra I and II and Geometry 
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should have the knowledge base they need to earn a score on the ACT in math sufficient for 
admission to many colleges. 

On the other hand, some incongruities do exist. Oklahoma students and teachers would be well 
served if the discrepancies were identified and addressed. After all, it is in the interest of both 
students and the state for high school students to score as high as possible on the ACT. To be 
specific, the Standards for Transition include conditional probability, the solving of absolute 
value equations and inequalities, properties of special right triangles, composite functions, 
circular functions, and Venn diagrams as a specific counting technique, and these topics are not 
explicitly addressed in PASS. Conversely, PASS includes work with transformations in several 
algebraic and geometric contexts, which are not explicitly addressed in the Standards for 
Transition. Oklahoma’s standards are also much more explicit in their emphasis on the concept 
of slope. In addition, differences exist in the treatment that the two sets of standards give to 
geometry. 

One solution might be to define and assess a rigorous core set of high school math expectations, 
as a number of states have done. This core would describe what is required for students 
graduating from Oklahoma schools to be successful as citizens, consumers, employees and 
college students rather than what students need simply to land entry-level jobs. While this core 
would be independent of course selection, it would include topics already in the PASS set of 
expectations for Algebra I, Geometry and Algebra II. It should be crafted at a level so that 
students who complete only Algebra I are able to earn a high school diploma but students who 
complete higher-level math enhance their chances of receiving higher scores on the ACT and 
qualifying for a broader range of worthwhile careers. This sends the right message to students: 
Chances for success in the adult world are enhanced by completing a high school math program 
based on the rich contents of PASS, including most of the skills listed for grades 9 through 12. 

Ultimately the proof of the compatibility of the two sets of standards will be in the comparability 
of the assessments. Students, teachers, parents and the general public will judge the intent of the 
standards by the types of assessment items that appear on local and state tests as well as on the 
ACT. Oklahoma’s assessments will need to be thoughtfully constructed to maintain many of the 
rich aspects of PASS standards. If this occurs, these assessments will necessarily have a 
somewhat different flavor than the ACT test. Care will need to be taken in test development to 
ensure that students who pass state assessments will be able to score at a level on the ACT that 
permits access to higher education opportunities in Oklahoma. 
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Major Findings: Mathematics Alignment of Tests to Standards 

Reviewers examined four assessments administered to Oklahoma students. Three of these tests 
are part of the Oklahoma School Testing Program (OSTP) while the fourth — the ACT 
Mathematics Test — is supported by the Oklahoma State Regents for Higher Education as a tool 
to help prepare high school students for postsecondary work or school. The tests examined were 
as follows: 

• Grade 5, Oklahoma Core Curriculum Test, Multiple Choice Test Booklet (2002) 

• Grade 8, Oklahoma Core Curriculum Test, Multiple Choice Test Booklet (2002) 

• End-of-Instruction Algebra I, Oklahoma Core Curriculum Test, Form 1, Field Test 

( 2002 ) 

• ACT Mathematics Test, 97-98, Form 57B 

Each of these assessments was examined using Achieve’ s assessment-to-standards protocol. For 
the purpose of this study, the tests were compared with the appropriate grade-level or course- 
level standards contained in PASS, as revised in May 2000. 

Standards used for this alignment study were as follows: 

• Grade 5 Mathematics Content Skills and Mathematics Process Standards for Grades 1-5 
(Grade 5 Test) 

• Grade 8 Mathematics Content Skills and Mathematics Process Standards for Grades 6-8 
(Grade 8 Test) 

• Algebra I Mathematics Content Skills and Process Standards for Grades 9-12 (Algebra 1 
End-of-lnstruction Field Test) 

• Algebra 1, Geometry and Algebra 11 Mathematics Content Skills and Process Standards 
for Grades 9-12 (ACT Mathematics Test) 

All of the assessments reviewed consist entirely of multiple-choice items. Math tests for grades 5 
and 8 — developed and published by Harcourt Educational Measurement — consist of 50 items 
each. Five items in each test are designated as field-test items, so student, school, district and 
state scores are calculated based on a total of 45 items per test. The suggested time for both tests 
is 55 minutes, but students may be given extra time — up to one additional hour — if needed. 

No calculators are allowed during these tests. 

The End-of-lnstruction Algebra 1 field test — developed and published by CTB/McGraw-Hill — 
consists of 80 multiple-choice items. Students are given approximately 100 minutes to complete 
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this test, which is being administered for the first time — as a field test — in spring 2002. They 
may not use any resource materials, including calculators, during the assessment. The purpose of 
this test is to measure the algebra proficiency of students who have completed an Algebra I 
course. Oklahoma students must take this test, but need not pass it, to graduate. 

The ACT Mathematics Test — intended for students in grades 1 1 and 12 — consists of 60 
multiple-choice items, and students have 60 minutes to complete this test. They are permitted to 
use calculators, although all problems on the test can be solved without using calculators. Most 
four-function, scientific and graphing calculators may be used, and examinees are responsible for 
ensuring their calculators meet ACT requirements. The content areas assessed on this test 
include: Pre-Algebra, Elementary Algebra, Plane Geometry, Coordinate Geometry, Intermediate 
Algebra and Trigonometry. 

The following summary highlights and explains the most important findings from the study of 
the alignment of Oklahoma Core Curriculum Tests and ACT assessment to Oklahoma standards 
in math. More detailed findings are included in the accompanying secure, technical report. 

Strengths of the Assessments 

Oklahoma Core Curriculum Tests in mathematics generally align well with the content 

and performance expectations of Oklahoma’s standards. 

Oklahoma provided Achieve with test blueprints that show how the state matched individual 
items on each test to a state standard. Reviewers foimd they had substantial disagreements with 
the maps provided by the state for grades 5 and 8 and, consequently, remapped about 30 percent 
of the items on these two tests. (They had fewer concerns with the mapping of Algebra I items, 
disagreeing with the state only about 7 percent of the time.) Remapping had three effects: One 
positive effect is that alignment scores improved, since reviewers were able to find a closer 
match for the items in question to Oklahoma standards than did the state; however, on the 
downside, the revised maps indicate that some items actually assess standards at previous grade 
levels and show fewer items mapping to algebra topics and more to topics that fall in the 
category of number sense and operations. Using the revised maps, reviewers found that 
Oklahoma’s Core Curriculum Tests show a strong degree of alignment with PASS for math; all 
items are either strongly aligned or partially aligned to standards. 

Oklahoma’s Algebra I field test is on the right track; it has an appropriate level of rigor 

and is generally well balanced — fairly representing the knowledge and skills described 

in the standards. 

The field test is technically well crafted with the result that no items were eliminated from 
consideration because they were flawed in one way or another, such as having two or no correct 
answers or problematic graphics. For the most part, items have a good span in terms of 
intellectual demand. While the cognitive level of the field test is primarily at the recall and basic 
problem-solving level (applying skills and concepts), the proportion of problem-solving items far 
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exceeds those rated as recall items. And 5 percent of items require students to reason, plan, use 
evidence and employ a higher level of thinking. 

Although, on the whole, the field test is balanced, it does emphasize Data Analysis, Statistics and 
Probability more than one would anticipate from reading the standards, given that only four of 
the 23 content skills are in that strand. An imbalance in one strand inevitably precipitates an 
imbalance in one or more other strands. In this instance, the strand that suffers is Functions and 
Relations; it contains 1 3 of the 23 content skills, many of which are content-rich, being 
composed of multiple parts. On more than one occasion, items mapped to such compound 
content skills assess just one of the subskills multiple times, leaving others not assessed at all. 

Areas for Improvement 

Oklahoma should improve the accuracy of the test blueprints. 

As previously noted, test blueprints provided by the state are not as accurate as they should be, 
given the importance of these maps in judging the alignment of tests to standards and their effect 
on score reports that are sent to schools. Reviewers disagreed with the way the test developer 
mapped 14 of the 45 non-field-test items (30 percent) on the grade 5 test and 13 of the 45 non- 
field-test items (29 percent) on the grade 8 test; they had fewer disagreements with the map the test 
developer provided for the Algebra I field test and remapped only six of the 80 items (7 percent). 

Having test blueprints (or maps) be as accurate as possible is important for two reasons; The map 
is the basis both for evaluating the degree of alignment of items to the standards on a test and for 
reporting patterns of achievement in the various content strands, such as Algebraic Operations. If 
a significant number of items are misassigned and reviewers adhere to the state blueprint, they 
would rate these items as not being well matched to the standards, and alignment would be 
negatively affected. The second effect of mismapping is that patterns of student results may be 
misleading. For example, what appears to be a strong showing in Algebraic Operations can 
disappear if the majority of the items mapped to this strand are found, on closer examination, to 
actually assess Number Sense. 

Oklahoma should raise the level of rigor of the Core Curriculum Tests in mathematics 
at grades 5 and 8. 

The level of rigor of Oklahoma’s tests in grades 5 and 8 is generally too low. Three major factors 
contribute to the lack of intellectual challenge. One is the fact that reviewers judged 
approximately 20 percent of the items on each test as being more appropriately mapped to 
content skills at earlier grade levels. (Although not identified as a major flaw, reviewers also 
noted a tendency for items on the 5*-grade test to assess the easier concepts or the less 
cognitively demanding performances as expressed in compound content standards.) 

A second factor is that both tests place a heavy emphasis on recall and procedural skills at the 
expense of the more cognitively demanding skills typically called for in application and problem 
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solving. For example, no items on the Oklahoma 5*' or 8‘^-grade tests were scored by reviewers 
as level three (strategic thinking), meaning they require students to reason, plan, use evidence, 
and employ a higher level of thinking than do levels one and two. (Because of the extended time 
period typically required for level-four items, which involve complex reasoning, planning, 
developing and thinking, the inclusion of such items in a large-scale, on-demand assessment is 

rare.) However, it is possible, and desirable, for a large-scale assessment — even one made up 
entirely of multiple-choice items — to include level-three items. 

A third factor is Oklahoma’s sole reliance on a multiple-choice format for its assessment items, 
which tends to overly limit the kind of performances a test is able to measure. It is certainly the 
case that test developers can construct multiple-choice items that address students’ abilities to 
understand abstract concepts, recall and apply simple algorithms, test their skills in reasoning, 
and solve relatively complicated problems. Oklahoma does not push the limits of its multiple- 
choice questions far enough. Oklahoma may wish to consider augmenting its tests with short- 
answer and open-response items. Short-answer items can tap a different level of skill than can 
multiple-choice items and can assess some content more efficiently than open-response items. 
For example, a short-answer question can require students to generate the appropriate equation 
for solving a problem, rather than recognizing and selecting it from a list of four choices. Open- 
response items have additional assessment power — they provide students the opportunity to 
demonstrate a deeper and more sophisticated understanding of math by asking them, for 
example, to construct graphs, write algorithms or equations, and explain their answers. These 
kinds of items also provide a better venue for assessing student ability to integrate and make 
connections across different area of math for example, the relationship between algebra and 
geometry. Equally important is the impact variable test formats have on instruction. The 
judicious use of variable formats can enhance the rigor of a state’s tests, but its real impact is its 
effect on the way teachers teach and construct classroom assignments and assessments and the 
way in which students prepare for tests. 

ACT is not strongly aligned to Oklahoma’s high school course standards in 

mathematics. 

Items on the ACT generally address the content and performances described in PASS; however, 
10 of the 60 items on ACT math standards map to standards at grade 8 or earlier. This is not 
surprising given that tests like the ACT are meant to compare student performance on a broad 
span of knowledge and skills and not meant to certify individual student performance as basic, 
proficient or advanced, as standards-based tests do. 

Reviewers called attention to another aspect of the ACT that does not align particularly well to 
Oklahoma’s standards in math, and that is the tendency of the ACT test to favor a procedural 
approach to problem solving over reasoning. While this fits well with the multiple-choice nature 
of the ACT, it does not conform to the expectations articulated in Oklahoma’s standards. 

While the ACT does test knowledge and skills found in Oklahoma’s standards, it only tests a 
portion of the knowledge and skills that Oklahoma describes in its standards. In fact, to be 
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specific, less than 40 percent of the content knowledge in Oklahoma’s courses in Algebra I, 
Geometry and Algebra II is assessed by the ACT in math, with the poorest coverage occurring in 
the content area of Data Analysis, Statistics and Probability. It is worth noting, in this regard, that 
the ACT test in science reasoning, which is part of the overall ACT assessment package that all 
students take, does contain some items that assess some areas of the Data Analysis, Statistics and 
Probability strand, but not aspects of the strand that address mathematical modeling. 

Reviewers do not recommend that Oklahoma change its standards to conform more closely to the 
ACT test in math; Oklahoma’s standards are more broadly conceived, as they should be. Given 
the significant role that the ACT has historically played in helping to raise standards and increase 
the proficiency of Oklahoma students in math and its high profile in the state, it makes sense for 
the state to point out clearly to teachers and students where the ACT includes mathematical 
topics not contained in PASS standards. As noted earlier these include complex numbers, 
composite functions, special triangles and solving absolute value equations. 
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Expert and Staff Biographies 

Achieve relied on the expertise of nationally respected experts in academic content, standards, 
curriculum and assessment design to inform and conduct the standards benchmarking and 
alignment of assessments to standards. 

Achieve Senior Project Directors 

Matthew Gandal 

Matthew Gandal is executive vice president of Achieve, Inc. He manages the Washington, D.C., 
office and is responsible for overseeing Achieve’s major initiatives. These include the 2001 and 
1999 National Education Summits and a series of follow-up activities Achieve has launched to 
help states address the Summit challenges; the Benchmarking Initiative, which helps states 
compare their standards, assessments and accountability policies with those of other states and 
nations; the Mathematics Achievement Partnership (MAP), which is designed to help states 
improve curriculum and instruction in middle school math and measure student achievement 
using a common, internationally benchmarked S^’-grade test; and the American Diploma Project. 

Before joining Achieve, Gandal was assistant director for educational issues at the American 
Federation of Teachers (AFT). He helped AFT launch a variety of programs and publications 
designed to support standards-based reform efforts in states and school districts. He was the 
author and chief architect Making Standards Matter, an annual AFT report evaluating the 
quality of the academic standards, assessments and accountability policies in the 50 states. He 
also authored a series of reports that compared student standards and achievement in the United 
States with those of other industrialized nations. 

Gandal, a graduate of the Maryland public school system, earned a bachelor’s degree in 
philosophy from Trinity College in Hartford, Conn. 

Jean Slattery 

Jean Slattery has been a consultant for Achieve since 1999 and currently serves as director for 
the Benchmarking Initiative. She was supervising director of curriculum development and 
support in Rochester, N.Y., from 1989 to 1997, with responsibility for overseeing the work of all 
subject-area directors in the K-12 instructional program. Her earlier responsibilities as a district- 
level administrator included serving as director of the middle school (1987-89) and junior high 
(1985-87) programs. During this period, she initiated Teachers as Partners, a peer-coaching staff 
development program funded by the Ford and Matsushita (Panasonic) Foundations. 

Slattery also is a peer consultant on standards and assessment for the U.S. Department of 
Education. She has served as a consultant to the Washington, D.C., school district; San Diego 
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Unified School District; a Washington state consortium of mral schools; and the Alabama and 
Illinois Departments of Education. Slattery has also worked for the Council for Basic Education 
on projects involving the Flint Community School District, the Nevada Education Department 
and the Cleveland Municipal School District. 

Slattery received a bachelor’s degree in chemistry from Albertus Magnus College, a master’s 
degree in science education from Yale University and a doctorate in science curriculum from the 
University of Rochester. 

English Language Arts 

JoAnne Thibault Eresh 

JoAnne Thibault Eresh is a senior associate at Achieve, where she leads the English language 
arts aspects of the Standards-to-Standards Benchmarking and Assessment-to-Standards 
alignment reviews. She taught writing at the university level and English at private and public 
high schools in St. Louis, Mo., and in Fitchburg, Mass. She began her work in curriculum design 
and performance assessment in 1979 under Superintendent Richard C. Wallace Jr. and from 
1981 to 1994 was director of the Division of Writing and Speaking for the Pittsburgh Public 
Schools. During that time, she directed The Pittsburgh Discussion Model Project, funded by the 
Rockefeller Foundation and part of the CHART network, and later directed the imaginative 
writing part of the ARTS Propel Project, a joint project with Harvard’s Project Zero and the 
Educational Testing Service. She was the Pittsburgh district coordinator for the New Standards 
Project and wrote the teachers’ guides for the New Standards ELA portfolios. In 1995, she was 
one of the original resident fellows at the Institute for Learning at the University of Pittsburgh’s 
Learning Research and Development Center, as well as coordinate the New Standards Linking 
Projects. From 1997 to March 2001, she was the coordinator of staff development in Community 
District Two in New York City, where she was responsible for the hiring, training and 
coordinating that district’s staff development group. 

Standards Reviewers 

Arthur N. Applebee 

Arthur N. Applebee is a professor of education at the State University of New York at Albany. 
Since his seminal history of the teaching of English in 1974, Applebee has been nationally 
recognized as a leading authority on English language arts. His studies focus on how children 
and adults learn the many specialized forms of language required for success in school subjects, 
life and work. In 1998, he received the David A. Russell Award for Distinguished Research in 
the Teaching of English from the National Council of Teachers of English for his book 
Curriculum as Conversation: Transforming Traditions of Teaching and Learning, a 
reconceptualization of the role of curriculum in American schools and colleges. Applebee has 
also examined the development of story telling and story-telling skills among children. He has 
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experience in program evaluation, high school teaching (English and drama), and clinical 
assessment and treatment of children with severe reading problems. 

Applebee received his doctorate from the University of London. 

Susan Pimentel 

Susan Pimentel, co-founder of StandardsWork®, a nonprofit education consultancy, specializes 
in standards-driven school reform and works as an education writer, analyst and coach. After 
earning a bachelor of science degree in early childhood education and a law degree from Cornell 
University, Pimentel served as senior policy adviser to Maryland Governor William Donald 
Schaefer and subsequently as special counsel to former Superintendent John Murphy in Prince 
George’s County, Md., the nation’s 16‘*’-largest school district. 

In recent years, her work has focused on helping communities and schools throughout the 
country work together to advance meaningful and enduring standards-based education reform. 
This includes the development and implementation of rigorous grade-by-grade standards, results- 
based evaluation systems, diagnostic assessments and a powerful new reporting tool, called The 
Results Car(i^, which helps communities and educators stay focused on student achievement. 
StandardsWork focuses on building the system from the inside out, equipping school leaders 
with the resources and support they need to sustain the process of continuous improvement, close 
community collaboration and data-driven results. 

Beyond her work with districts, Pimentel has emerged as an expert in standard setting. The 
Fordham Foundation has ranked the states in which Pimentel has coordinated the standards- 
setting effort as having among the best content standards in the country. California and Arizona 
were ranked first and second, respectively. She has also helped Maryland revise its English 
language arts and social studies standards, raising that state’s overall rank from 43'^'^ in 1998 to 
lO'*’ in 2000. Pimentel is co-author with Denis P. Doyle of the best-selling book and CD-ROM, 
Raising the Standard: An Eight Step Action Guide for Schools and Communities. 

Assessment Reviewers 

Sheila Byrd 

Sheila Byrd is project director of the American Diploma Project, an innovative multistate 
initiative of Achieve and three other education organizations designed to ensure that states have 
set the bar for high school exit requirements in the right place. Before starting her own education 
consulting firm, she was deputy director of California’s Academic Standards Commission, 
established to develop academic standards for the state’s K-12 public schools. She previously 
had served the commission as its senior consultant for the English Language Arts Committee. 
Prior to her appointment to the Academic Standards Commission staff, Byrd was the 
administrator of the Education Leaders Council (ELC), a national network of state 
superintendents and state boards of education members. 
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A former educator, Byrd’s other professional activities include work as a consultant on the U.S. 
Department of Education’s frequently cited Third International Mathematics and Science Study 
(TIMSS) and as a liaison to the California Commission on Teacher Credentialing’s RICA 
Advisory Panel. She currently is an appointee of the California State Board of Education to its 
SAT 9 Augmentation Panel, where she is helping create the state’s new standards-based K-12 
assessment. 

Sue Craig 

Sue Craig is a retired educator. She hais been an outreach educator for the Oregon Museum of 
Science and Industry and a classroom management trainer for both the California Teachers 
Association and the Oregon Education Association. Craig is a member of the California State 
Board of Education State Assessment Language Arts Content Review Panel, a member of the 
California State Board Performance Level Setting Panel, a member of the California State Board 
panel to develop writing prompts and scoring guidelines, and a member of the California 
Teacher Credentialing panel on reciprocity. 

A classroom teacher for 30 years, Craig’s other professional activities include teaching special 
education, U.S. history, social science, English and English as a second language. 

Eunice Greer 

Eunice Greer is the division administrator for the Illinois State Board of Education’s Illinois 
Reads-Statewide Reading Initiative. In recent years, Greer has led the Standards and Assessment 
Division of the Illinois State Board of Education and been an assistant professor in the 
department of curriculum and instruction of the University of Illinois, the director of research for 
the Harvard Graduate School of Education, and the literacy assessment coordinator for the 
University of Illinois’ Center for the Study of Reading. 

She is a member of the American Educational Research Association, Association for Supervision 
and Curriculum Development, International Reading Association, Kappa Delta Pi, National 
Council of Educational Measurement, National Council of Teachers of English, Phi Delta Kappa 
and Phi Kappa Phi. 

Greer was awarded the Excellence in Teaching award by the University of Illinois in summer 
and spring 1997; fall, summer and spring 1996; and fall 1995. She also was awarded the 
Distinguished Contributions to the New Standards Project award in 1995 and the Illinois 
Reading Council’s Annual Achievement Award in 1993. She received a master’s degree and 
doctorate in education from the University of Illinois at Urbana-Champaign. 

Laura McGiffert 

Laura McGiffert is director of the Mathematics Achievement Partnership (MAP) at Achieve. 
MAP is a multistate collaboration to dramatically improve math performance in the middle 
grades. As the principal manager of this project, McGiffert assumed primary responsibility for 
the development of Foundations for Success: Mathematics for the Middle 
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Grades, which represents the core knowledge and skills that students should learn to prepare for 
high school and beyond. To this end, she coordinated Achieve’s Mathematics Advisory Panel, an 
expert panel of mathematicians, math educators, curriculum specialists, and state and local math 
supervisors representing a broad spectrum of perspectives about math education. 

Before joining Achieve in 1998, McGiffert was a high school English language arts teacher for 
five years in Colorado, where she was involved in a districtwide effort to refine and align local 
standards and assessments. In 1995, she was awarded Best First-Year Teacher in Eagle County 
School District at the high school level. She also taught writing and literature at Colorado 
Mountain College. 

A native Washingtonian, McGiffert holds a bachelor’s degree in English and American literature 
from Harvard University, a master’s degree in education policy from the Georgetown University 
Public Policy Institute, and a master’s degree in secondary education from the University of 
Colorado at Boulder. 

LeRoy Miller 

LeRoy Miller is in his 28‘^ year of teaching 1 l‘^-grade English at Sidney High School, where he 
also serves as English department chair for a nine-member department. Miller also is a teacher of 
Advanced Placement and honors American literature and an adviser for the Academic 
Competition Team. In addition, he served as a commissioner on the Governor’s Commission on 
Student Success in 2000 and is a member of Ohio’s Academic Content Standards Writing Team 
for English language arts. Miller received a bachelor’s degree in English from Fairleigh 
Dickinson University and a master’s degree in secondary English from Miami University. 

Mathematics 

Kaye R. Forgione 

Kaye R. Forgione began consulting work with Achieve in 2000 and joined Achieve as senior 
associate for mathematics in March 2001. Forgione’s primary responsibilities are managing and 
providing intellectual leadership to Achieve’s standards and benchmarking work involving math. 
Before joining Achieve, she served as assistant director of the Systemic Research Collaborative 
for Mathematics, Science, and Technology Education (SYRCE) project at the University of 
Texas at Austin. Her responsibilities also include administrative and design responsibilities for 
UTeach, a collaborative project of the College of Education and the College of Natural Sciences 
to train and support the next generation of math and science teachers in Texas. Prior to her work 
at the University of Texas, Forgione was director of academic standards programs at the Council 
for Basic Education, a nonprofit education organization based in Washington, D.C. 

Prior to joining the Council for Basic Education in 1997, Forgione worked in the K-12 arena in a 
variety of contexts including district-level curriculum supervisor for math, assessment and 
gifted/talented programs. She also was team leader for assessments and task development and 
supervisor in the areas of assessment, school profiles and educational indicators at the Delaware 
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Department of Education; senior research associate for development at the Delaware Education 
Research and Development Center at the University of Delaware; and a high school math 
teacher. Her personal portfolio of work includes math-related and policy development work in a 
number of states (including Nevada, Maryland and Hawaii) and school districts (including 
Cleveland; East Allen County, Ind.; and Los Angeles) and in partnership with a number of 
organizations (including Achieve, George Washington University and the Institute for 
Educational Leadership). Forgione earned a doctorate from the University of Delaware. 

Standards Reviewers 

Susan K. Eddins 

Susan K. Eddins has taught students in kindergarten through college for more than 30 years — 

24 years at the high school level. She is the recipient of several honors for her teaching, 
including the Presidential Award for Excellence in Mathematics Teaching, and she is a National 
Board Certified Teacher in Adolescent. and Young Adult Mathematics. Eddins is a faculty 
member, an instmctional facilitator, and the curriculum and assessment leader in math at the 
Illinois Mathematics and Science Academy, where she has taught since the school’s inception in 
1986. 

She has served in leadership capacities in several professional organizations and is currently a 
member of the board of directors of the National Council of Teachers of Mathematics (NCTM). 
Eddins holds bachelor’s and master’s degrees in mathematics. 

Eddins was a member of the 9-12 writing group for NCTM’s Principles and Standards for 
School Mathematics. She is co-author of a chapter in NCTM’s Windows of Opportunity and is a 
co-author of UCSMP Algebra. She is a past panel member and editor of NCTM’s Student Math 
Notes and has authored several articles in refereed journals. Over the past four years, in addition 
to numerous workshops and presentations, her most extensive work has been in the area of 
standards development, standards review and alignment of standards to assessments. For 
Achieve, she has reviewed academic standards or assessments from Alaska, Illinois, Indiana, 
Minnesota, New Jersey, Oregon, Pennsylvania, Texas and Washington. 

R. James Milgram 

R. James Milgram is a mathematics professor at Stanford University in California. He also has 
been a visiting professor at the mathematics institute in Barcelona in June 1 998, at Northwestern 
University in March 1997 and at the University of Montreal in May 1995. 

Recent lectures include those given at Santa Cmz University, University of Chicago, University of 
Illinois and Indiana University. Milgram, together with Gunnar Carlsson, Ralph Cohen and Steve 
Kerckhoff, revised the California Mathematics Standards and Framework under the direction of 
and for the California State Board of Education. He also was a member of the content review panel 
for math curricula for the California Textbook Adoption in 2000 and has been a member of the 
National Science Foundation Panel on Mathematics and Robotics since May 2000. 
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Milgram is a member of the Mathematics Achievement Partnership (MAP), a project of Achieve, 
Inc., to create a common set of expectations and assessments for math in the middle grades, and 
was an expert math reviewer for Achieve’s study of Texas’ proposed objectives and the Texas 
Essential Knowledge and Skills (TEKS) student expectations for the second Texas Assessment 
of Academic Skills (TAAS) study. 

He received a bachelor’s degree in science and a master’s degree in science from the University 
of Chicago and a doctorate from the University of Minnesota. 

Lynn A. Steen 

Lynn A. Steen has been a member of the St. Olaf College faculty since 1965 and currently is the 
director of institutional research and planning. He is leader of the quantitative literacy initiative 
of the National Council on Education and the Disciplines (NCED) at the Woodrow Wilson 
Foundation and a member of the Mathematics Achievement Partnership (MAP), a project of 
Achieve, Inc., to create a common set of expectations and assessments for math in the middle 
grades. Steen is former president of the Mathematical Association of America and former chair 
of the Council of Scientific Society Presidents. He has reviewed math standards for Achieve 
from Massachusetts, Oklahoma and Texas. He received his doctorate in mathematics from the 
Massachusetts Institute of Technology. 

Assessment Reviewers 

Susan K. Eddins 

Donald R. King 

Donald R. King is an associate professor of mathematics at Northeastern University. Previously, 
he was a visiting assistant professor at Salem State College, a visiting assistant professor at the 
University of California at San Diego and a high school math teacher in Boston. King is a 
member of the Mathematical Association of America, American Mathematical Society and the 
National Association of Mathematicians. King is active in professional and community service: 

He was a parent member of the Mathematics Focus Group for Boston Public Schools in 1997; 
director from 1993 to 1994 of NUMATH, Northeastern University’s program to foster minority 
mathematical achievement and talent in high school; an adviser to algebra-in-middle-schools 
projects from 1990 to 1992; a review panelist for three years for Ford Foundation postdoctoral 
fellowships for minorities; and an adviser to Massachusetts’ pre-engineering program for 
minorities from 1988 to 1991. King recently gave a speech at the American Mathematical 
Society’s Special Session on Teaching Mathematics in the New Millennium titled “Changing 
school outcomes: raising standards and promoting equity,” and he has advised Achieve on the 
quality of standards and assessments in a number of states including Minnesota, New Jersey, 
Oklahoma and Texas. 

King received his doctorate in mathematics from the Massachusetts Institute of Technology. 
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Mary Lynn Raith 

Mary Lynn Raith currently is a mathematics specialist in the Division of Instructional Support of 
the Pittsburgh Public Schools. As such, her responsibilities include leadership roles in curriculum 
development; textbook selection; design of alternative assessments — both performance tasks 
and portfolio development; in-service program design and implementation; in-class support of 
teachers; and coordination of math programs across levels and schools. She has special 
responsibility for middle schools: Raith is the co-director of the Pittsburgh Reform in 
Mathematics Education project (PRIME), a K-12 professional development system that includes 
both in-class support from demonstration teachers and a comprehensive series of in-service 
sessions focused on increasing the depth and breadth of teachers’ math knowledge. 

Prior to this position, Raith was a math supervisor (1986-96) in Pittsburgh and a middle school 
math specialist in grades 6 through 8 (1970-86) working with remedial as well as gifted students. 
She has designed and presented — at local, regional and national conferences — sessions on the 
infusion of algebraic thinking, geometric reasoning, statistics and probability, and problem 
solving in the K— 8 math program. In summer 1987, Raith was chosen to attend a Michigan State 
University (MSU) honors teachers workshop and since then has been involved with the 
implementation, piloting and in-servicing of MSU programs, including the Middle Grades 
Mathematics Program in Pittsburgh and the Connected Mathematics Project in Pittsburgh and 
other school districts across the nation. 

She has been involved with a number of national projects, including the Assessment 
Communities of Teachers project, which supported middle school teachers in the use and 
development of assessment tools in their classrooms, and the Alternative Assessment in 
Mathematics project (A^IM), which defined criteria that identified adequate progress in math 
achievement for Title I students. 

Raith was a middle school leader for the New Standards Portfolio project, which designed a 
portfolio system for use in both classroom and district assessment. She also was a middle school 
leader for the New Standards Reference Examination development project, which included 
heading up the task and rubric-development, scoring and standard-setting teams. In partnership 
with the New Standards math team she has presented at math leadership conferences in New 
York; Washington, D.C.; and Bucks County, Pa. Raith joined the National Council of Teachers 
of Mathematics (NCTM) Academy faculty and has presented the Principles Academy atNCTM- 
sponsored national conferences. She also has worked extensively with the National Center on 
Education and the Economy (NCEE) and the America’s Choice school design on designing and 
delivering professional development on standards-based math curriculum, instruction and 
assessment and has presented at the annual NCEE national conferences. Raith received a 
bachelor’s degree in mathematics from Indiana University at Pittsburgh and a master’s degree in 
mathematics education from the University of Pittsburgh. 
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